Tales from the Wonderside

Disclaimer: I am an OFW. I do recognize our part in keeping the Philippine economy running. I know the hardships and sacrifices our kind undergo, working far from home and family, having to work in alien cultures and, on some cases, suffering discrimination and abuse. I know the importance we play in keeping it all together for our country. That our remittances saved the nation time and time again, and without it our economy would probably be in shambles right now.

Let me define the context of the word "hero" first. I refer primarily to a patriot. Somebody with the awareness and deep connection to his nationality, almost bordering jingoism. Someone who has done a meaningful sacrifice, one that cannot be easily duplicated. That sacrifice has to be done with a specific goal in mind: that to the benefit of the community or the nation. 

I do remember trying the greener pasture in a neighboring country not out of a sense of sacrifice for my country, but rather the simplistic notion of getting a bigger paycheck. I don't wish to apologize for it, certainly not to glamorize, but nonetheless admit that it was utterly selfish when compared to what those I consider heroes should aspire to be. Not entirely selfish since a major part of it is my willingness to provide for my family. Being the bread winner for my retired parents, plus wife and newborn, I think my decision was entirely justified even for just the motive of financial survival.

The thought of doing it to better our economy was non-existent. I know and have heard of the tales of OFW's ever since I was a kid. The songs and commercials and movies they made to glorify this particular worker type. Not that there's many of those types of media anyway but the appreciation from the government was there. Was it a lip service or genuine patronage is of no consequence in this article.

Nor have I met anybody do it for the purpose I just described. It was all the same - there is a company in a foreign land willing to pay for my services at a higher rate. Sending those hard-earned cash back to the homeland will somehow triple their value, thanks to the exchange rate. Globalization at its finest, and they think they should take full advantage of it. This is the psychology of a would-be OFW. I bet every single OFW out there will unanimously agree on this.

Now, is that heroic? Maybe for our families. Though, as a father, that is my responsibility. The only difference between working on another country and back home is the pay and proximity to loved ones. Other than that, it's just plain work - the very promise of capitalism we are all trying to get by with the hope of a comfortable retirement one day. It just the same as going from the provinces to Manila. It's almost the same as the daily commute we endure to get to our offices or work sites. It's an eventful act of separation from our comfy beds so we can pay rent and provide food to the table.

Let me paint a picture of heroism that I think is a real representation and has to be aspired by the youth. It has nothing to do with a bigger paycheck. It may contribute to a blissful retirement one day. It does separate from loved ones but not with a great distance. The heroes I refer to are the public school teachers. I know teaching is the noblest of professions, but they do it on an entirely different leve. Low wages, 60 students per class if you are lucky, decades old equipment, amongst other things. One could argue that they do it for the wages anyway and not for the love of kids or the country, but there is one event some of them show this is not truly the case. I haven't seen this done by an OFW nor anything that remotely resembles it.

I speak of national elections. They have the fortunate additional task of doing the actual electoral process, from hosting the voting centers, to manual counting to distribution of ballots and the like. Now, elections in the Philippines, especially in rural areas, are not as orderly as in other developed nations. They are full of sensationalism, violence and is mostly a popularity contest. Owe it to the corrupt culture where candidates are usually vying for position in order to steal money to recoup their campaign expense. Most importantly, owe it to the voters who do not know better.

Some of these teachers have faced gunpoint from corrupt candidates wanting to take the ballots for whatever scheme they intend for their victory. Many of these teachers refused then at the cost of their lives. All of them are too underpaid to really protect these pieces of paper with their lives, much less to be responsible for this civic duty in the first place. Those that died, did so because they believed in the power of the ballot and the importance of elections. Whether elections in the Philippines actually represent the opinion of the majority aka the masses, or whether they really make progress for the country is worthy of another discussion. This particular act of sacrifice presupposes a level of thinking that takes into consideration an abstract notion of a democracy; something they can never really feed their families with nor guarantee them an actual retirement. A really high level of consciousness indeed. Heroes in every angle, national heroes, worthy of respect and admiration.

Now, contrast this with a typical OFW, like me. I believe to some degree that, given a chance to switch citizenship to a more developed country most Filipinos would do so without second thought. Especially if they can bring over their family and have a job waiting for them on the new land. I don't see anything wrong here - many people from any civilization from any point of human history have done this thing. It's a natural human trait. Entirely justified.

Is that a heroic attitude? 

I stil love my country, and would like to go there every now and then. This is mostly due to the idea that there's no place like home, even if your home is a poorly-run joke fest of a republic. I still want to wear my Filipino pride everywhere I go, but if there's a slight chance for a better life somewhere else for my family I will definitely give the latter a go. 

Contrast our sacrifices to that of public school teachers especially during election time. Contrast that to the soldiers fighting our fellow countrymen in the South. Contrast that to the honest and hard working employees of the civil service. 

I am not belittling the conditions that some other OFW's are undergoing right now. The psychological burden of being away from family, especially your children can never be put into words. Add to that the possibility of one day seeing your child, whom all your hard work is dedicated to, no longer recognizes your face. 

I'm simply saying that these unsung heroes need more spotlight. We call OFW's modern heroes but there are a lot of heroes in our country. If you intend every child in our country to aspire to be an OFW we will not only experience an even worse brain dran but an actual exodus, where none are actually willing to come back given the chance. We need heroes that have attained a level of consciousness that goes past the rudimentary everyday worries of a salaried worker. We need icons of real patriotism rooted in hard work and a genuine love for their community. If only we can vote for public school teachers.

My current company just concluded our inaugural online competition for local school kids, and there had been quite a handful of lessons learned with regards to making things scale for a simple online competition. I spearheaded the development and was provided support by my superb teammates, and the biggest stumbling block of all is about dealing with database gotchas. 

    I chose MongoDB because it seemed very lightweight and the ease of changing the models. I could live without joins - I found them very slow anyway. Data consistency can be enforced by ensuring simplicity in how your models interact, and since our project was projected to be a lightweight application (it is), I know I can pull it off as long as I keep things small and simple. 

Being an online competition for school smarts the stack was relatively straightforward:

Tornado as the web server and also serves static files 

Nginx for load balancing. Now I think I should have gone for Nginx to solely serve static files though.

Node.js game server using Socket.io for message passing between game clients as well as the multiplayer queue

MongoDB as the database

Then came the height of the competition where the top leader spots are being cemented, and then the mistakes and lack of familiarity became more apparent. I guess I was too used with cloud services that I've began to neglect best practices in database setup. This was compounded by the fact that our awesome backend guy is also working with MongoDB for the first time. Suffice to say, we all had our share of surprises.

Here is a short list of things to remember when trying out MongoDB the first time for production use, in no particular order:

Use an asynchronous library. I used PyMongo behind the Tornado web server, and while Tornado was fast as advertised and PyMongo is reliable I should have used an async library like Motor (a fork of PyMongo). I think the reason for this choice is just me being intimidated by callback hell.

Create your indexes as you develop. This makes lookups faster and you can be very flexible contextually on which fields to use. This applies to other databases as well.

Shard from day one. If you expect your application to scale then sharding should be part of the initial database setup. This also helps support #2 since indexes take up space. Sharding also assumes a replica set, which will help bring in secondaries when your primary goes down.

I have done reading Steve Job's biography by Walter Isaacson and have been spotting several reactions from bloggers of different backgrounds almost every week. There doesn't seem to be a consensus on what really made Steve Jobs successful, and the varying opinions would always either extrapolate or deviate from the supposed Reality Distortion Field. I could also see many so-called product people putting their worst behavior towards their peers, on varying levels, in the attempt to emulate the product development success. This trend seems to have stemmed not from his death but rather the biography, as the only approved account of his life, both personal and professional.

I must admit that I had the temptation of letting go of my inhibitions and simply just say what I have in mind, regardless of how it will affect other people, be it at work or at home. It came from the hope that by being brutally honest and vocal I can replicate such a success. Add to that a binary view of the world where something is either shit or is the shit, and what we have is a persona that pushes away people most of the time.

I also had the pleasure of watching the House MD series right from the first season, as well as the occassional re-run of the Iron Man movies. Here the parallels are striking: some guy with amazing talent or expertise on a very narrow subject matter is allowed to have his way and be an asshole provided he delivers. The behavior exemplified is easily childish and egotistical; the hubris and the drama makes for a great story, just like what I've read in the biography. A technologist or expert that struggles with his inner demos and tries to put up the arrogance mask all the while saving the day and making the right moves when it counts. It's one thing to have those kind of personalities in fiction; it's another thing to have a living proof albeit an outlier.

Back to Steve. What really struck me on many of his stories, both in the book and from other accounts, is his seemingly overwhelming awareness of himself. I know he can't be totally honest with others as most people are, but he seems to know exactly what he wants which somehow pushes the right buttons on people to agree with his agenda. This may sound like a rewording of the Reality Distortion Field but I really think there is nothing to distort; it is a given that us humans exercise this great power of shaping our environments and our fortunes by sheer will, and to do so may imply a herculean effort but isn't necessarily supernatural or even mystical. He was constantly shaping and building, which came from his passion of marrying art with technology. This existentialist conclusion concludes that nothing in life is set in stone, and that everything can be twisted and sculpted - it's always just a matter of time.

This kind of mindset somehow echoes that of the Zen attitude towards life and reality, at least from what little I understood. I knew Steve practiced a plethora of Eastern disciplines especially as a young adult, Zen Buddhism included, and being part of the hippie movement made him prioritize his spiritual walk over many other practicalities in life. His spiritual search, I think, led him to the realization that he is virtually alone in life, and that really deep down he is nothing special; this allowed him to work things from scratch and direct his time and energy to things that really matter to him. Other people being swayed by him is simply a by-product of having a clear vision of where he needs to go as a person. The products he helped build are a testament to the clarity of purpose and the design considerations that runs through the mind of somebody that has seen the future; not in a mystical sense of the word but simply wanting something to happen and believing it because everything else in life is just some other people's wish coming true.

By being committed to his vision he didn't have rationalize so many things, especially when running Apple or overseeing the development of a certain product. There is a mental picture, and all he has to do was execute. This obviously is really nothing new, and we have thousands of stories like this, not to mention that this happens on a daily basis. The success of Apple as a company has to be seeded by this, although there are many other factors that contributed to it, mostly through other people which is out of Steve's control. I bet anybody who has this clear vision of the future and a drive to seeing it into fruition can achieve success in his own endeavor. It may not be in the scale of Apple but if we are to care about dichotomies winning is winning - to achieve one's goals is just as productive as completing as self-imposed task. The kind of stamina and faith required to achieve such things had to be inherent in most of us.

I forgot to spread this around so I might as well post it here. I have created a mailing list for local node.js users/hackers/enthusiasts:

http://groups.google.com/group/node-js-philippines/

Admittedly, I didn't go for the usual intensive introspection routine I usually do when the year turns. Instead, I saw the end of 2011 as having a lot of unfinished business and new insights that I feel I need to see come into fruition. That being said, here's what I plan to do (and not do) for the rest of the year.

This is not a new year's resolution list; this is more of a concrete plan on what I need to focus on, at least on the first part of the year.

Blog more

Most of my blog posts have been on the technical side, and I realized that since this blog is also my site carrying my name as the domain I might as well open up with the other facets of my life. Granted, it isn't much, but there are still stuff that can be shared and moments to capture. In short, more frequent posts with more diverse topics.

Learn a new programming stack.

I made a survey on what could be both challenging and beneficial to me as a freelancer, and I settled on IOS development using Objective-C. I passed on the other choices (server-side Javascript, RoR, Haskell etc.), which I'll still be touching from time to time.

The initial readings proved quite a challenge coming from Python - I think I was babysitted by such an expressive language that going back to my C roots introduced some sort of a culture shock; or maybe it's just that Python grew into me and now I think more like a Pythonista for using it everyday.

This will be quite a ride, and the end game would be to publish in the App Store that can supplement my income.

Complete the Stanford online class I signed up 

I was a bit disappointed with how my Machine Learning stint came about - I didn't find the whole process of working with data very engaging. I could change my mind in the future but somehow I don't have the statistician blood in me.

So when Stanford offered a slew of online classes for 2012 I signed up to all, with the intent of downloading the video lectures for future reference and focusing on just one class. The class I've chosen is the Human-Computer Interaction, which focuses heavily on interface design and development. I would leverage my knowledge here when writing IOS apps so I could relate more with users and do a better job than what I did with CVStash. 

I could also use another class as a second focus, but I have yet to see that with my current schedule.

Prepare for my wedding.

I'm tying the knot with my long time girlfriend, and I need to prepare a bunch of stuff (legal, financial, ceremonial, etc.) to make sure we have a memorable moment come summer time. I have most things laid out because I was initially planning on a short civil wedding last December, but my significant other wanted more formality and we agreed halfway so we went for a ceremony with quite a handful of guests come summer 2012.

Why I included this here is because such commitments do not start nor end on the wedding night. I wish to make my transition to the married life a smooth one, especially in terms of schedule. Relationships and family takes time and effort, and what I do to make it worthwhile must coincide wtth my other goals so happiness can be complete all throughout.

Get back into shape

I have almost completed a vertical jump program during our village' version of a league "lockout", mainly due to flooded streets. I have yet to notice any change in my jump but during the training I shed some pounds and felt good throughout the day because of the intense workout. I plan on restarting the regimen, but I could probably supplement it with simple yoga exercises for more flexibility.

I also wish to do running around the village and in the C6 highway, but that may not be too feasible. I have yet to work on a schedule to accomplish this.

Brush up on my maths

This will be an ongoing struggle, having quite a stigma for mathematics most of my youth but actively worked on getting the simplest of concepts back into the limelight the past year. I owe a lot to Khan Academy, and I plan to do a weekly review and lesson using the site. I plan to rework my Calculus by summer and start to be comfortable with Linear Algebra by the end of the year.

  Conclusion

I've been working on getting myself familiar with IOS development, and I went on to learn how to make simple games. The open source project Sparrow looked to be a good start, but I encountered some problems getting it to run on the first try. Here's what I did to go around it:

Make sure you have git installed.

Clone the project's master branch, which should contain the latest stable release: git clone git://github.com/PrimaryFeather/Sparrow-Framework.git

Run the demo in Sparrow-Framework/samples/demo/src/Demo.xcodeproj

If you run into workspace errors, close XCode as well as the IOS Simulator and reopen the Demo project. 

Once you can play around with the demo, setup the Sparrow source tree in XCode. Go to XCode Preferences > Locations > Source Trees tab. Add the entry SPARROW_SRC pointing to where you have the sparrow src/ folder (mine is /Users/arbiesamong/ios/sparrow/Sparrow-Framework/sparrow/src). Make sure there are no spaces in the path. You know it's the correct folder if you can see the file Sparrow.xcodeproj

Now we must verify if the scaffold runs. Close all XCode and IOS Simulator windows.

Copy Sparrow-Framework/samples/scaffold/ to a directory of your choosing.

Open scaffold/src/AppScaffold.xcodeproj 

In the Project Navigator pane, click on the AppScaffold project.

Verify that the Sparrow subproject, Sparrow.xcodeproj is in blue. This means that the Sparrow source can be found. If not, try to set the Location in the Utilities pane on the right to "Relative to {{ sparrow_source_tree_name }}". 

Try to run the project. You should see a red square over a black background when the simulator loads the code.

Sets in Python

Python defines a set as a list with unique elements. The function set() creates such sets, and one can create from a list by doing

mylist = [1,2,3] myset = set(mylist) myset # set([1, 2, 3]) type(myset) #

If there are duplicate elements in #mylist#, the duplicates are discarded leaving only one instance of such an element. Furthermore, the resulting set will be sorted automatically.

We also know that strings in Python can also be treated as a list of characters although they are not equal in terms of type. This similarity is highlighted when we compare a set derived from a string versus a set of a list of characters:

'abc' == ['a','b','c'] # False set('abc') == set(['a','b','c']) # True

Traditional set operations are also available:

set1 = set(['m','a','k','a','t','i']) set2 = set(['m','a','n','i','l','a']) set1 - set2 # set(['k', 't']) aka xor set1 | set2 # set(['a', 'i', 'k', 'm', 'l', 'n', 't']) aka union set1 & set2 # set(['a', 'i', 'm']) aka intersection set1 ^ set2 # set(['k', 'l', 'n', 't']) aka disjoint

Membership in a set can be queries pretty much like how one does with lists:

22 in set([1,11,22,33]) # True 'sam' in 'arbie samong' # True 'bes' in 'arbie samong' # False

Lastly, we can make comparision operations between sets:

Disclaimer: I'm a VIM guy for a year, and a GEdit guy for years. I don't blog about it because I use it everyday, and everyday stuff is just part of me so I don't feel it needs extensive writing at the moment.

Somehow things changed when I got my Macbook Air, and the Escape key is no longer pleasant to work with. So I'm now in the process of trying another editor for daily use, and none of the IDEs available really give me a kick in terms of utilizing finger speed. So I'm writing down a list of the most basic of Emacs commands to memorize in case I forgot.

First off, I found emacs alredy bundled in the MBA, and accessible in the terminalv via the emacs command. Also, you'd want to make full use of the alt key, so you go to your Terminal > Prefrences > Keyboard and check "Use option as Meta key". These commands were given in the built-in tutorial inside Emacs, accessible via Ctrl + h, t (push the Ctrl key, hold it, hit h, release, then hit t). Now for the commands:

  Screen navigation

Ctrl + v View next screen

Alt + v View previous screen

Ctrl + l Clear screen and center to cursor

Alt + < Move to beginning of file

Alt + < Move to end of file

  Cursor navigation

Ctrl + n Next line

Ctrl + p Previous line

Ctrl + f Move cursor forward one character

Ctrl + b Move cursor backward one character

  More cursor navigation

Alt + f Move cursor forward one word

Alt + b Move cursor backward one word

Ctrl + a Move to beginning of line

Ctrl + e Move to end of line

Alt + a Move back to beginning of the sentence

Alt + e Move back to end of the sentence

  Basic utility commands

Ctrl + u <n> <item> Execute item n times. Item can be command or character.

Ctrl + g Cancel current command

Ctrl + z Suspend Emacs; actually a UNIX command. Type fg in terminal to resume.

Ctrl + x, Ctrl + c Exit Emacs

Deleting

Backspace (or delete in MBA keyboard) Delete previous character

Ctrl + d Delete next character

Alt + backspace Kill (aka Cut) previous word

Alt + d Kill next word

Ctrl + k Kill to end of the line

Alt + k Kill to end of sentence

  Copy + paste

Ctrl + Space Start marking for kill (Mark set)

Ctrl + w Kill from Mark set to current position

Ctrl + y Yank aka Paste

Alt + y Yank with previous kill (hit repeatedly to cycle through previous other kills)

  Undo

Ctrl + x, u Undo

Ctrl + _ Undo

  Files

Ctrl + x, Ctrl + f <filename> Find (open existing or as new) a file

Ctrl + x, Ctrl + s Save

  Buffers

Ctrl + x, Ctrl + b List all buffers

Ctrl + x, b <buffer> Switch to buffer

Ctrl + x, 1 Close other buffers

Ctrl + x, s Save some buffers (has confirmation)

  Windows

Ctrl + u, 0, Ctrl + l Mark start of bottom window display

Ctrl + x, 2 Display new window where cursor is placed on marked start

Ctrl + Alt + v Scroll down bottom window page

Ctrl + o Switch windows

Ctrl + x, 4, Ctrl + f Open file in new window (has prompt for filename)

Ctrl + x, 1 Kill other windows

  Recursive Editing

Alt + x Get into a mini buffer

Esc, Esc, Esc Get out of mini buffer and close all windows

  Editing modes

Alt + x, <mode> Change to this mode

Ctrl + h, m Show help on current major mode

Ctrl + x, u, <n>, Ctrl + x, f Set margin to n characters

Alt + q Refill the paragraph under the Auto Fill minor mode

  Search and replace

Ctrl + s Search (has prompt)

Alt + x, replace-string Replace string (has prompt)

  Getting help

Ctrl + h, c, <command> Show short help on a command

Ctrl + h, k, <command> Show long help on a command

Ctrl + h, f, <function> Show help on a function

Ctrl + h, a <string> Show all commands that match string

Ctrl + h, i Read online manual

We'll continue our exploration of linear regression, and now we'll be discussing about the gradient decent algorithm that will help us find the minimum for our cost function. Most of what's discussed here including the graphical media are courtesy of Stanford University's online Machine Learning class: http://ml-class.org. It's free to join so I suggest you go there and sign up if you wish to learn Machine Learning from the experts. My posts will just serve as personal notes plus my interpretations, though in some ways this can be seen as a tutorial of sorts. The original material and a better lecturer by the person of Prof. Andrew Ng can be found on the machine learning class site.

Gradient descent is a general algorithm that helps us arrive at some minimum value for a function. This can be used in our cost function to find the best hypothesis that will fit our training set. Note that gradient descent is not just used in cost function scenario but in other applications as well; basically those that needs to find a minimum.

In our case, we have our cost function J(Ɵ0, Ɵ1) and we basically want to get the minimum for this. Since gradient descent is an algorithm (not a formula), we'll do good to describe how it goes:

Start with some (Ɵ0, Ɵ1). This will be our arbitrary starting point to kick off the algorithm. Usually we can start with Ɵ0=0 and Ɵ1=0, but this could actually be any starting point.

Keep changing the values (Ɵ0, Ɵ1) to reduce our cost function J(Ɵ0, Ɵ1), until we end up with a minimum value.

Simple, isn't it? :D

Now, let's visualize for a moment our cost function graph which will now come in 3d since we will not be assuming Ɵ0=0. This means we have 3 variables, the cost function J(Ɵ0, Ɵ1), and then Ɵ0 and Ɵ1.

You'll notice that this looks more varied than the bowl-shape we have the last time. This is just a helpful visualization, but for linear regression problems this graph will stay in its bowl-shaped figure.

The nice thing about this graph, though, is it looks like a typical hillside scene. I'm not trying to just get your imagination running here, as this metaphor is actually *the* basis for gradient descent.

Remember, our goal here is to arrive at a minimum point for our function. Using the 3d graph, we can imagine the low points to be those that are in blue and are obviously situated lower on the vertical axis. Using gradient descent, we then pick some arbitrary starting point (the violet dot):

Now, imagine that you are that starting point. You'll see that you are in a place in the hill, and you basically want to go down to the lowest point. So you take a 360-degree look around and think of what direction to step to. You take a baby step down, and then check again for the next step. You do this until you get to the lowest point aka the algorithm converging to the lowest point.

We'll call the lowest point you have arrived to be the local minimum. This is in contrast with the global minimum, which is the lowest point for the whole graph. The local minimum is just the lowest point nearest to your starting point as deduced by the gradience descent algorithm. We need to make this distinction because if we pick a different starting point (even if it is near the original starting point), we will end up with a different lowest point aka local minimum:

Here we see the green dot starting off just a bit to the right of the original purple starting point, and it finds a different lowest point that is quite far from the original. This shows that the local minimum will largely vary from the given starting point, and this is a property of gradient descent. Another thing about the local minimum is that, if a value like Ɵ1 has already reached it, the gradient descent algorithm will leave Ɵ1 unchanged on its next iteration.

Now we can express this idea mathematically. We describe the gradient descent algorithm to be:

Let's take this apart:

We need to perform the operation descibed in between the curly brackets until convergence aka a loop.

Ɵj holds the current value.

We are using :=, meaning an assignment operator, in contrast with the = sign, which is a truth assertion. The symbol := basically says that we assign the value in the right to the variable in the left.

The α character is called the learning rate or the alpha value of the algorithm. It describes how big are the steps to take per iteration. If alpha is too small, the descent will be too slow. If it is too big, it can overshoot the minimum and go overboard. This latter case could cause the algorithm to fail to converge, or even diverge.

The rest of the right hand side of the assignment is a derivative. I'll explain this more in a while.

The statement in parenthesis on the right is a subtlety in gradient descent. It says that we need to update Ɵ0 and Ɵ1 simultaneously. To describe more, here is a comparison of the correct way of updating Ɵ0 and Ɵ1 to the wrong way:

All this is saying is that Ɵ0 and Ɵ1 both need to be tracked of their values for the given iteration. Their values will only get updated as we enter the next iteration. The 'incorrect' way descibed here forms yet another algorithm altogether, which provides a very different behavior from gradient descent and is not covered here. To give a quick illustration, if Ɵ0 = 1 and Ɵ0 = 2, if we apply gradient descent for just one step/iteration we will get:

Ɵ0 := Ɵ0 + sqrt(Ɵ0*Ɵ1)

Ɵ0 := 1 + sqrt(1*2)

Ɵ0 := 1 + sqrt(2)

and

Ɵ1 := Ɵ1 + sqrt(Ɵ0*Ɵ1)

Ɵ1 := 2 + sqrt(1*2)

Ɵ1 := 2 + sqrt(2)

which clearly shows how we used the values of Ɵ0 and Ɵ1 simulataneously, aka we don't update the values until the next iteration.

Now let's discuss a bit about the derivative part of the algorithm:

This simplest explanation for what is this is that it describes a slope that touches the point for the current value of whatever parameter it has. To illustrate this graphically, let's assume Ɵ0 = 0, and we only have Ɵ1. Suppose our cost function graph looks like the usual parabola, and we picked a certain arbitrary value for Ɵ1:

Since gradient descent will update the value of Ɵ1, the derivative part of the algorithm is the slope to which the value of Ɵ1 coincides with the line of the function, like this:

We can also see this as the slope to which the value of Ɵ1 must "travel" in order to reach the minimum. Since we have our slope on the right side of the parabola, we say that our derivative is a positive number. If it is on the left side, then it will be a negative number, as it is a negative slope.

Another property of this derivative is that if we are at the local minimum, the value of the derivative is zero, since the slope will be a flat horizontal line.

The derivative is important because it describes the behavior of the algorithm as it reaches the minimum. As we approach the local minimum, gradient descent will automatically take smaller steps. This makes it unnecesary to decrease the learning rate over time as this is taken care of by the derivative. This means that the derivative actually controls how much the learning rate progresses throughout the iterations. This, in effect, makes the algorithm take smaller steps over time:

And basically it's this derivative term that enables the gradient descent algorithm to actually minimize the value of our cost function. This also means that the algorithm can coverge even if the learning rate is kept fixed (though the learning rate must not be too large). Here is our algorithm beside the linear regression model we discussed before:

So, in order to achieve the minimalization process using gradient descent, we need to change our derivative to fit the cost function:

Here we simply substituted J(Ɵ0, Ɵ1) with our partial derivative term and also spelled out our hypothesis function. Now we can apply this to a single step in our gradient descent algorithm, with Ɵ0 and Ɵ1 as our parameters in the context of our hypothesis function we get:

If we put those two together we get our modified gradient descent algorithm which now actually reflects what we are trying to do with our cost function:

So let's see the algorithm in action. Here we have the familar graph of our hypothesis function against the training data on the left, and on the right our cost function represented as a contour figure (since it now in 3d: J(Ɵ0,Ɵ1), Ɵ0 and Ɵ1).

Next, we pick some arbitrary value to start with. It is usually in (0,0) for (Ɵ0, Ɵ1), but now let's start off with (-900, -0.1).

Suppose we take one step to the left, which also changes our line for our hypothesis function:

Then another step. Notice how the line of our hypothesis function changes on the left graph:

We keep adjusting the point in our cost function graph on the right until we get to the local minimum (also the global minimum here since the contour graph translates into a 3d bowl-shaped figure called a convex function, which is just the 3d equivalent of a parabola). This is important as all linear regression problems actually don't have a local minimum and just one global minimum. As we get to the local minimum we see our hypothesis function fit more nicely with our samples:

Now we can make our prediction. If we have a house around 200 square feet we can sell it at around $150k to $170k.

It is worth noting that there are other version of gradient descent that behave differently from the one we described here. This version of gradience descent is technically called Batch Gradient Descent. The batch word alludes to the fact that we take into consideration all the training samples in our calculation, as seen in our formula for our cost function where we take a summation. We will talk about those other versions as we progress through our study.

We'll continue with our discussion of linear regression for our Machine Learning series. This also serves as notes for my participation in the Stanford Machine Learning class online (http://ml-class.org). You can also join this class as it is free and generally self-paced.

As a review, we learned about linear regression which is an algorithm that lets us map values over some input using a function we call the hypothesis. The hypothesis function is a linear function, and the best hypothesis function for our training data is that which can accomodate the training data in the best way mathematically possible. And in order to determine that best way, we use the tool called a cost function and minimize it (an optimization process) so we can arrive at the best possible hypothesis for our data.

The cost function helps us determine the best way to fit the line in our data. By itself the cost function would simply describe to us the range of possible values for our slope that could fit our training set, so that is not what we need. What we really want is to minimize the cost function, so that the line that we get best accomodates for every sample that we have. Formally, we denote the cost function as J(Ɵ0 + Ɵ1x).

In our formula, hƟ(x) = Ɵ0 + Ɵ1x, the cost function will help us determine the two paramters, Ɵ0 and Ɵ1. Since this is a function, it maps nicely to the slope intercept form y = mx + b, which simply means that m = Ɵ1 (aka the slope) and b is Ɵ0 (aka y-intercept or x). To illustrate this further, consider these simple values for these two terms:

The first figure has a zero slope and the y-intercept in 1.5, the second has a zero y-intercept and a slope of 0.5, and the third having a slope of 0.5 and y-intercept of 1. In this case, this very function is our hypothesis and is responsible in making predictions from our data using linear regression.

To understand the role the cost function plays further, let us see an example. Here we have exactly three samples that constitute our training set:

Here we see the three samples in (1,1), (2,2) and (3,3) respectively. Remember, the goal of the cost function is to give us the best line that will accomodate our training set, which is accomplished by finding the minimum value for the function. In order to make things simple, we shall assume that Ɵ0 = 0, so that in our hypothesis function we are left with Ɵ1 to tinker with.

This also means that our line always passes the origin (0,0) with the just slope (Ɵ1) being different from example to example. Since we are trying to understand this as a function, let us try different values for Ɵ1.

We shall start with Ɵ1 = 1. This should give us a nice upward slope that's easy to plot.

Since Ɵ0 = 0, our the value of the hypothesis function will simply be Ɵ1x. The value for the second term, y^(i), would simply be the vertical distance from the slope to the sample (or the y value). Now, let's apply the cost function formula here. We now have sufficient information to solve for the cost function when Ɵ1 = 1.

So for Ɵ1 = 1, the cost function is equal to 0. Let's keep track of the values of our cost function using a graph:

Don't forget that this is the graph of our cost function. This is different from the graph of our hypothesis function. We'll collect values of our cost function using this graph. Let's try out Ɵ1 = 0.5:

And the formula computation being:

We should notice here that each squared term (hƟ(x) + y) is just the height of each sample against the hypothesis line. This will also apply even if Ɵ0 != 0. So the first term is (0.5 - 1) because hƟ = 0.5 and y = 1. You should clearly see now how the other terms came about. The result of the formula is 0.58.

Suppose Ɵ1 = 0, we'll have a flat line on our x-axis:

Which should give us the value 2.3 if we applied the formula like this:

Now let's plot these in our graph containing cost function values.

We can plot more possible values, and it should be obvious to some that positive values would also have negative counterparts, and vice-versa. That means that if we plot more values here we will definitely get a parabola:

So here we have the graph of the cost function, but what we're really interested in is the minimum value. And we have an obvious winner:

That is, when J(Ɵ1) is 0 and Ɵ1 is 1. Now, the interesting part is that this minimum value somehow actually fits the training set perfectly!

This is because the cost function, as the name suggest, gives us an idea of the cost or margin of error for our hypothesis function against the training data. Which is why we are after the minimum value, which means we want to minimize the cost or margin of error. This way we know which particular hypothesis function actually fits our training data as far as univariate linear regression is concerned.

Up until now we still assume Ɵ0 = 0. If we do consider a nonzero value for Ɵ0 we'll obviously introduce another dimension for our cost function graph, and the 2-dimensional representation is no longer sufficient. Needless to say, it will be a convex figure, but this time it does look like a real bowl:

This is because we are now plotting the cost function with Ɵ0 != 0. As with the original 2d form, the minimum value is of course at the bottom of the bowl.

We can still use a 2d figure but we will employ what is known as contour figures, which assumes that Ɵ0  and Ɵ1 are of the same level, giving a 2d illusion:

We can still extrapolate the bowl shape from here, like it is growing from the middle. The bottom of the bowl, which also has the minimum value or the set of hypotheses that better correspond to our sample data, can be found in the smallest circle.

Remember that Ɵ0 is just the y-intercept, which means that in our hypothesis graph (still in 2d) we are just raising the line vertically. We can imagine different combinations of Ɵ0 and Ɵ1 that will fit the sample data better, and for mathematical precision we use the cost function to determine which is the best hypothesis function to use. To recap, here are the formulas we've encountered so far:

Linear regression is a type of a supervised Machine Learning algorithm that can be used to predict some continuous value based from certain input(s). We knew from the last post that supervised learning is when a "right" answer is provided in your training data, and regression probolems are supervised learning problems where you need to predict continuous values (as compared to discreet ones).

An algorithm works this way. We start with a training set, or data that we can deduct from with our algorithm. We then feed this data into our learning algorithm, which will produce us a function traditionally called the hypothesis. The hypothesis function will now be able to interpret future input into output based from what it learned from our training data.

A good example here is when you are trying to predict a good price for your house based on its size, say in square meters. Now, usually you can just pick the price off your head but we want to be precise, so we will base it on sale prices of houses on your area based on their sizes.

 We can imagine one input, the size of houses, and the output being the price of the house. This can also serve as the training data for the algorithm. Since we are just working with two dimensions, it will make is easy for us to work with this mathematically if we are to plot this in a cartesian graph:

We placed the input (size) in the x axis, and the output (price) on the y-axis. The red dots represent the a house price given the size, denoted by (x,y). In tabular form we have:

So we can see some example values for sizes (x) and prices (y). Now we are showing 4 samples, which we denote as m = 4. We can, of course, have as many samples as we want, and usuallythe more the better.

Since we are talking about one input, bieng the size of the house, and we need to predict a continious value output being the price, we can say that this constitutes a univariate (single-variable input) linear regression problem.

We say linear because our algorithm simply assigns a linear function to be the hypothesis. Visually, we can also imagine a line plotting the hypothesis function in our graph.

Where we say h(x) simply maps values of x to an estimated value y. The formal notation for h(x) is: hƟ(x) = Ɵ0 + Ɵ1x. Hint: this is just a fancy way of expressing the slope-intercept form.

Setting up a Django environment on Windows XP

My desktop has a dual boot setup where I do real work in Ubuntu and the games go in XP. Recently, some weird @$@# happened in my Ubuntu install and I can't access the Interwebs there for some reason. deadlines are looming so I got desperate and had to setup on my XP so I can work my Django projects there.

Imagine that.

Now here's a summary of what I did to get me up and running. 

Download Python

Get it here: http://www.python.org/getit/

  Setup paths

Right click on My Computer > Properties. In the System Properties window, click on the Advanced tab. Then, in the Advanced Section, click the Environment Variables button.

In the lower part of the window, you'll see the System Variables pane. One of the entries would be the Path. Click on Edit. 

Append the path of your Python executable here. I downloaded Python 2.6, so mine was c:\Python26\. To cater for most situations, you need to also add the scripts folder. Here's the general form of the path string you need to append:

c:\Python26\;c:\Python26\Scripts\

  Setup virtualenv

First, grab setuptools:

http://pypi.python.org/pypi/setuptools#downloads

This will give us the easy_install.exe program in our c:\Pythonx.x\Scripts\ directory. Then, fire up the command prompt and enter:

$> easy_install virtualenv

Once we have virtualenv, we can now use it similarly to how we do in Linux. As for me, I'm a big virtualenvwrapper fan, so here's a very helpful script that somehow emulates that for us:

http://justindriscoll.us/2009/03/virtualenv-helper-for-windows.html

I named the script in my system workon.py, so it closely resembles that of virtualenvwrapper. 

I also prefer pip when grabbing packages, so better get it now:

$> easy_install pip

  Test Django

Create a virtual environment and a project folder anywhere in your system, say c:\django\

$> env.py test_project

$> mkdir c:\django\

$> cd c:\django\

Install Django with pip.

$> pip install Django==1.3

That should add the relevant django-related executables in the Scripts folder. Try out the startproject command:

$> django-admin.py startproject mysite

And do runserver:

$> cd mysite

$> python manage.py runserver

Machine Learning made a sudden resurgence due to the successes of many web companies and projects that employ it, and from recent strides in Artificial Intelligence research. It can be said that the current wave of Artificial Intelligence work is resting upon the successes of Machine Learning developments.

We see more and more of it touch our daily lives, from web search engines to filtering email spam, to making purchase recommendations and even intelligent machines. This makes it worthy of study even for a layman like me. I shall try my best to give a gentle introduction to this fascinating subject and hopefully encourage others to spend time knowing more about it too.

  Definition of Machine Learning

There had been many definitions of Machine Learning throughout the years, and researchers/practitioners have yet to agree on it. A popular definition was given by Arthur Samuel in 1959:

Field of study that gives computers the ability to learn without being explicitly programmed.

The context for his meaning was that Samuel was a novice checkers player, and he observed that noting which positions lead to winning over losing (in a span of thousands of games) will eventually lead a computer to conclude which of them give the most probability for winning. At the end of that exercise, the computer program actually performed better than Samuel.

A more recent and rigorious definition was given by Tom Mitchell in 1998:

A computer program is said to learn from experience (E) with respect to some task (T) and some performance measure (P), if its performance on T, as measured by P, improves with experience E.

So basically what Mitchell is saying is that programs that learn do so by gaining experience and we, as observers, can see an improvement on its performance (based from a defined set of performance measures) over a certain task.

In contrast with traditional programming, we only have the programming task (T) and the performance measure (P), the latter we usually attribute to testing. The experience (E) factor is gained usually (and hopefully) by the programmers and not the programs themselves. Here we have programs acquiring experience when performing a task, and when we apply our measurements to such a performance we would conclude that they are actually doing better than the last time.

  A Machine Learning problem

To give an example, consider the case for filtering spam emails as a basic machine learning problem. Suppose we have a program that takes note of emails we mark as spam, and based on that learns how to filter future emails as such. We can then analyze this simple example and conclude that:

The task (T) is to classify whether an email is spam or not

The performance measure (P) is the number or fraction of emails correctly identified as spam or not

The experience (E) would be watching you label emails as spam or not

It is also important to note that these three factors need to be present and related in the exact same way as Mitchell described in order for a problem to be considered as a machine learning problem. There are many interesting problems in the world, and only a subset actually qualifies as a machine learning problem.

We have two (2) common types of machine learning problems, namely:

Supervised Learning

Unsupervised Learning

Basically, supervised learning is where we provide the criteria for what constitutes a quality output. It becomes supervised since we are providing what makes an output to be considered reliable, hence, "supervising" it. On the other hand, unsupervised learning is when no such criteria are given. We shall be discussing these two types in more detail.

  Supervised Learning

Supervised learning is basically a type of machine learning approach where the input data contains the "right answers" or value combination that we consider/interpreted as accurate. This way, when we try to predict values based off an algorithm, we can attribute a certain level of confidence to our prediction based from our input values, knowing that we have established our input values as "correct".

We have two (2) types of Supervised Learning problems:

Classification

Regression

Classification problems is where we are trying to predict discrete values from our input data. By discrete we mean a fixed set of values. An example would be to predict whether it is going to rain or not, and so we may considering the probable values to be just:

[ "will rain", "will not rain" ]

and these are a discrete set of possible outcome for whatever our weatherman decides to submit to us as weather data. Classification problems can have up to any amount of possible output discrete values. So given some weather data with lots of values for precipitation, wind speed etc., we shall come up with a prediction on wheter it will rain or not, and that is a basic example of a classification problem.

Regression problems are problems where we have to predict over a continuous set of values. By continuous we mean a huge set of probable output values, so we don't have a fixed set of output values but rather a whole system (like an integer output).

A good example for this is when trying to predict the possible prices for beef on the next season. A whole system of round numbers, including cents, is included in the range of possible output values, from Php 100 to a thousand and beyond. So we have different factors contributing to this situation, like current gas prices and the current population of cows in the supplier provinces, etc., and we have to predict based from these factors how much the price of beef will cost. Having this type of probable outcome makes this a regression type of machine learning problem.

  Unsupervised Learning

This type of machine learning problem is more commonly known as Clustering, since we don't have a criteria so we basically let our program to group or cluster the data as it sees fit. This work of clustering data provides the basis of Unsupervised Learning problems since we did not group or cluster them ourselves (criteria). 

A common example for this would be Google News. This site aggregates news pages all over the web and clusters or groups similar news under the same news heading. It does this by analyzing the keywords on a given news site and running them through Google's proprietary clustering algorithm, enabling it to present news as a cluster of web pages that talk about the same thing.

Another common example would be market segmentation, where we analyze customer data and run them through a computer program with the hopes of coming up with market segments (clusters) so we can better sell to each segment and strategize our business accordingly. We don't know beforehand what are the segments but we have data for customer behavior like demographics or the types of products that were purchased, so that makes it an unsupervised learning problem.

As far as I'm concerned by Machine Learning problem which will require you to cluster or group data will fall under Unsupervised Learning. 

  Conclusion

Machine Learning is indeed a fascinating subject to learn and to develop software with. The concepts discussed are pretty much what we encounter in our daily lives, especially if we spend way too much time on the Internet. 

The next set of discussions would be on learning more about the mathematical foundation of ML, which is very important if we want to create implementations that actually deliver. The topic would revolve heavily around Linear Algebra, more specifically Linear Regression. If you are not confident with your math-fu, then I suggest heading over to Khan Academy and brush up on it; I recommend revisiting basic Algebra, then Calculus, and then finally to Linear Algebra. This is important because some of the concepts would be very hard to communicate unless done through the beautiful language of Mathematics.

Also, as a first impression I noticed that the theoretical framework of Machine Learning borrows heavily from Statistics and Probability. Since these two subjects deal with making inferences based on input data, the same can be said of Machine Learning. It won't hurt to look into those subjects as well. I know I will.

This could also work for other Nvidia graphics card, just be sure to search for the appropriate drivers. You may also not need to switch to runlevel 3, but my driver installer asked for it.

  1. Download the drivers for your card here or just google it directly 2. Edit the blacklist conf file 

gksudo gedit /etc/modprobe.d/blacklist.conf

3. Add these lines then save+quit:

blacklist vga16fb

blacklist nouveau

blacklist rivafb

blacklist nvidiafb

blacklist rivatv

4. Uninstall existing Nvidia drivers

sudo apt-get --purge remove nvidia-*

5. Restart your computer as root console

6. Enable runlevel 3

telinit 3

7. A login prompt will appear, sign in with your normal account

8. Switch to root

sudo -i

9. Run the driver installer

sh NVIDIA-Linux-x86-190.42-pkg1.run 

10. Follow the instructions until completion.

11. Restart your computer with ctrl+alt+del

I just ran into a 500 Internal Server error with nothing on the logs whatsoever. After some digging it turned out to be a permissions problem, so first I did:

sudo chown -R www-data:www-data /home/username/public_html/*

Doing that will get rid of the error and show your page properly but you would need to sudo the hell out of editing your files. So I just gave my user the permission and set everything to 755 (this is on a dev server at my house):

sudo chown -R arbie:arbie /home/username/public_html/* sudo chmod -R 755 /home/username/public_html/*

Been experimenting with some Wordpress setups and ran into trouble implementing pretty permalinks, like:

http://localhost/myblog/this-is-a-single-post/

Turned out to be an Apache configuration problem. Here’s the solution that worked for my dev environment (Ubuntu Lucid Lynx):

First, make sure mod_rewrite is enabled:

sudo a2enmod rewrite

which adds the module. Checking with:

ls /etc/apache2/mods-enabled/rewrite*

you should get a single rewrite.load file.

Next, edit either /etc/apache2/httpd.conf or /etc/apache2/sites-enabled/mysite and change the option:

AllowOverride None

into:

AllowOverride All

Now, going back to Wordpress admin, go to Settings > Permalinks and set Custom Structure to /%postname%/. This should write to the .htaccess file automatically, or you could write it yourself. However, if you wish to have Wordpress do the editing of the .htaccess file for you because you’re a lazy irresponsible chump that runs his car into flooded streets, simply allow the webserver to write to the file:

chmod 755 .htaccess chown www-data:www-data .htaccess