The Coder's Way

I had been playing with ZeroMQ and I wanted to share it with you. First of all, it is not a complete messaging system such as RabbitMQ or ActiveMQ. A full fledged messaging system gives you an out of the box experience. ZeroMQ is not such a system at all; it is a simple messaging library to be used programmatically. It basically gives you a pimped socket interface allowing you to quickly build your own messaging system. This is very cool because you no longer need to use very complex monolithic systems, configure them, and have to have a dedicated system. It gives you sockets that carry whole messages across various transports like in-process, inter-process, TCP, and multicast. You can connect sockets N-to-N with patterns like fanout, pub-sub, task distribution, and request-reply. It's fast enough to be the fabric for clustered products. Its asynchronous I/O model gives you scalable multicore applications, built as asynchronous message-processing tasks. It has a score of language APIs and runs on most operating systems making it great for homogeneous environments. Specifically some benefits to using ZeroMQ are:

It handles I/O asynchronously, in background threads. These communicate with application threads using lock-free data structures, so ØMQ applications need no locks, semaphores, or other wait states.

Components can come and go dynamically and ØMQ will automatically reconnect. This means you can start components in any order. You can create "service-oriented architectures" (SOAs) where services can join and leave the network at any time.

It queues messages automatically when needed. It does this intelligently, pushing messages to as close as possible to the receiver before queuing them.

It has ways of dealing with over-full queues (called "high water mark"). When a queue is full, ØMQ automatically blocks senders, or throws away messages, depending on the kind of messaging you are doing (the so-called "pattern").

It lets your applications talk to each other over arbitrary transports: TCP, multicast, in-process, inter-process. You don't need to change your code to use a different transport.

It handles slow/blocked readers safely, using different strategies that depend on the messaging pattern.

It lets you route messages using a variety of patterns such as request-reply and publish-subscribe. These patterns are how you create the topology, the structure of your network.

It lets you place pattern-extending "devices" (small brokers) in the network when you need to reduce the complexity of interconnecting many pieces.

It delivers whole messages exactly as they were sent, using a simple framing on the wire. If you write a 10k message, you will receive a 10k message.

It does not impose any format on messages. They are blobs of zero to gigabytes large. When you want to represent data you choose some other product on top, such as Google's protocol buffers, XDR, and others.

It handles network errors intelligently. Sometimes it retries, sometimes it tells you an operation failed.

Quick note about running ZeroMQ on NodeJS. I tried to use npm to install ZeroMQ, and I had to make a few changes to the way ZeroMQ library was compiled on my mac OS X(10.6). You may have to change the warning level and install other libraries to get it working. Please contact me if you need help installing it!

ZeroMQ sockets are not the same as the sockets you know. When working with ZeroMQ, you need to think of them as magic communication hubs. Instead of thinking about the low-level socket and handshaking think about your message topology and flow. For example, you can have a ØMQ socket that binds to different ports and protocols at the same time. Your code will simply get the messages as if there is only one port and protocol. ZeroMQ also abstracts messaging patterns so if you want a ØMQ socket to act as a PUB/SUB or REQ/RES node you can just configure the ØMQ socket. Finally, ZeroMQ can inter connect and translate different protocols and messaging patterns so if you need to bridge a INPROC to TCP you can. Basically, ZeroMQ allows you to design your architecture and think about how you need all your systems to communicate with each other and not worry about how you are going to do it.

ZeroMQ handles all the complex work to make the messaging stable, fast, and reliable to let you focus on the fun part of architecting. The first abstraction in ZeroMQ is the transport layer. You can pick from a long list of transports. When designing your architecture ZeroMQ allows you to use 4 different transports:

INPROC an In-Process communication model

IPC an Inter-Process communication model

MULTICAST multicast via PGM, possibly encapsulated in UDP

TCP a network based transport

The TCP transport is often the best choice, it is very performant and robust. However, when there is no need to cross the machine border look at the IPC or INPROC protocol to lower the latency even more. The MULTICAST transport can be interesting in special cases. Finally, after you have picked your transport layer you need to decide how you are going to wire up the message passers and receivers notice how I did not refer to them as clients and servers. This is because ZeroMQ can be used to message pass between processes on the same box or over the network without a central dedicated server. ZeroMQ is a low-level library that you link into your code that lets you build complex messaging systems. You have to decide what code is going to be the server or client and think about what parts of the architecture is stable and what part is elastic. This is apart of what makes ZeroMQ so cool you as a programmer get to build your messaging system but, you do not need to understand the complex nature of making a robust mature messaging system.

ZeroMQ has a lot of options on how to connect the message queues up. In a later post, I plan to cover some of the different type of queues, but there is already a lot of really good documentation on their API at www.zeromq.org. The key idea you should walk away with is do you want to send a message out to many workers to do processing or you want to fan-in all the request so you will know when all the work is done. Do you want to use a pipeline so you can have serial processing or do you want to do something else. You can achieve many different designs by picking the right messaging pattern. It is amazingly easy to setup a ZeroMQ queue that already has the logic and transport to support your requirements. So you can just start sending a message to ZeroMQ and let it do all the magic of sending the data and receiving it.

A coworker and I were talking about the change in our industry and how it is under attack. He was very worried about how many disruptions are taking place like youtube, piracy, social marketing. He was frustrated that people do not understand that they are killing our industry and if we do not get paid for our content then they will never get any again. Specifically we were talking about the music industry, and I remembered a book I recently read about the history of the music industry. How short our memory is! The industry we talk about dyeing is the moguls like Sony, BMG, Universal, WEA, etc. The same guys that did the same thing to the live music venues a few decades a go.

That is right before the ability to record music the live venue's where the king of music and held all the power. Then recording and radio displaced the live venues and became the new kings. Now the Internet has displaced the moguls again. So why is this bad? Did we need them to get music? Not really what is going on is a pattern I am seeing in the business world. The cost of making music, distributing, and marketing is dropping. Basically the market and social efficiency has been found and the industry is crying why they can not have the old days back. The pressure they feel is the pressure of their own lack of efficiency. Ether they find new efficiencies or change the business model or they will just die. This is the natural cycle in business. Nothing lasts for ever, and the evolution of change drives new winners and losers.  That is why they have turned to the government and the Law to protect them. Typically the role the government and Law is to stabilize transitions and slow change and growth.

Durning the growth period, recorded music and radio found new resources and exploited them. No longer where their limitations on the number of people you could pack into the live venue or the number of stars that could be featured on the weekend. The job of being a musician exploded and the rat race was on. At the beginning of this cycle cost of marking, distributing, and marketing music was so expensive they're was a need for capital. If you think about the market place as a vacuum it pulls in all the needed resources to fill the void. Then everyone in the vacuum acts as a catalysts finding efficiency to fill it. But when you come close to the event horizon and find the maximum efficiency the ability to unfairly control and distribute resources ends. That is why the moguls are crying. At the beginning its easy to leverage and control the inefficiencies and profit off of it i.e. what the moguls do best.

Humans are amazing with their ability to find efficiency and exploit it. We also have a huge lust for acquiring stuff, physical or emotional. If you want to find the next big thing do not hang on to the end. Do not wait around, find the next cycle of birth so how do you do this? Its easy look for the same pattern. What is abundant and untapped? Like live venue they took a resource that was free the musicians that played for free singing in the fields and at home and created a market for them to play. The recording industry increased the market size and took the free resource i.e. the musicians and shared it with the people. Do you see the pattern? Find what is abundant and free that people already want and build a market with a different twist to it. This is the first step creating the vacuum then you need to become one of the catalysts finding the efficiency. If your lucky and one of the first to help fill the void you can profit off of it.

Everyone thinks startups are risky, that most don’t work, that they’re prone to failure. Actually, I find that working at a big company is what’s risky. Big companies tend to be more static; they don’t chase these new opportunities. They have a specific focus — they make turbines; they do television programming. They’re good at something and are very slow to adapt to new technology, while entrepreneurs and start-ups pick up on the new wave of technology and can be much more innovative.

What you’re involved in needs to scale, which means it needs to get cheaper every year and scale to millions of users. The company needs to be horizontal. You don’t do everything; you pick a slice that you’re good at. Like Intel — it doesn’t make computers; it just makes the chips that go into computers. And productivity is one of the most important things — it’s got to be productive. That doesn’t just mean being efficient; it means doing the right thing the right way.

We have JavaScript on the Browser, Mobile devices, Server, and now in the shell ;) what else can we ask for. JavaScript in our dreams? Maybe not but I really do love having one language to do all my work in. Not to long ago, I had to master a minimum of four languages to be productive. I would need to write my code in C#, JavaScript, PowerShell, SQL or PHP, JavaScript, BASH, SQL you get the point. Now with NodeJS I just need JavaScript to do it all. A few days ago, I needed to write a script that would create an account in the database, encrypt the password and link up all the permissions for that user across a few different tables. Then it would update some HTML based on rows added by the newly created user and push the code to the production server. It was easy as pie and came with all the benefits that the same code could run on my web server, command line, or cron job because its just a nodeJS package. In other words, its all JavaScript. I did not have to struggle to remember the crazy commands to do the work in BASH looking up man pages on sed/awk that I only use 4 times a year. It just is great to be able to use all the same tools you use every day. The same SQL client libraries, parsing tools, etc.

Writing a node shell script is easy it is no different from a Bash, Perl, or Python script. You just start the script with #!/usr/bin/env node and away you go! One hard thing with writing shell scripts is handling argument parsing and doing all the cool term outputs like progress bars, etc. You can just use packages like CLI or other NodeJS argument parsers to do all the heavy lifting so you can focus on writing code. I have started creating node shell script to do little tasks that I did manually. I am in love with my new workflow.

#!/usr/bin/env node var cli = require('cli') , crypto = require("crypto") , dbclient = require('mysql').Client , db = new dbclient() , clId; var options = cli.parse({ host: ['host', 'DB Host', 'string', '0.0.0.0'] , db: ['db', 'DB Database', 'string', 'database'] , dbuser: ['dbu', 'DB username', 'string', 'admin'] , dbpass: ['dbp', 'DB password', 'string', 'hummmmmm'] , username: ['u', 'New Username', 'string'] , password: ['p', 'Password', 'string'] }); var md5 = crypto.createHash('md5') .update(options.password).digest('hex'); db.user = options.dbuser; db.password = options.dbpass; db.host = options.host; db.database = options.db; db.connect(); db.query('INSERT INTO tableA (foo) VALUES(?)', [options.username], function(err, r) { if(err) return cli.fatal(err.message); cli.ok('INSERT INTO tableA >> ' + (clId = r.insertId)); db.query('INSERT INTO login (uid,pwd,clId) VALUES(?, ?, ?)', [options.username, md5, clId], function(err, r) { if(err) return cli.fatal(err.message); cli.ok('INSERT INTO login >> ' + options.username + ','+ md5 +','+ clId); // do other crazy work db.end(); }); });

For WeatherNation, I had to build an AJAX search box that would let users switch their current forecast zones. The requirements included searching citys, places, ICAO, zipcodes, and forecast zones. I also wanted to support additional signals for example if the user GeoIP was coming from colorado I wanted to rank Denver, CO over Denver, TN. This sounds like a great job for lucene or lets make it easier Solr. Solr is a wrapper around lucene and provides a very easy RESTFul interface. One of the hardest parts is setting up Solr so lets go do that now and get it out of the way.

$ cd /tmp/ $ wget http://apache.mesi.com.ar/lucene/solr/3.2.0/apache-solr-3.2.0.zip $ unzip apache-solr-3.2.0.zip $ mkdir /tmp/playground $ cd /tmp/playground $ cp /tmp/apache-solr-3.2.0/example/start.jar . $ cp -rp /tmp/apache-solr-3.2.0/example/etc . $ cp -rp /tmp/apache-solr-3.2.0/example/lib . $ cp -rp /tmp/apache-solr-3.2.0/example/webapps . $ mkdir -p data/weather/conf

At this point we have setup the most basic install of Solr. We are going to use start.jar that host the service in jetty and we have copied all the basic files need to run Solr. Lets update the basic configuration file. You can just replace the files with the version below.

# /tmp/playground/data/solr.xml <?xml version="1.0" encoding="UTF-8" ?> <solr persistent="false"> <cores adminPath="/admin/cores"> <core name="weather" instanceDir="weather" /> </cores> </solr> # /tmp/playground/data/weather/conf/solrconfig.xml <?xml version="1.0" encoding="UTF-8" ?> <config> <luceneMatchVersion>LUCENE_32</luceneMatchVersion> <directoryFactory name="DirectoryFactory" class="${solr.directoryFactory:solr.StandardDirectoryFactory}"/> <indexDefaults> <useCompoundFile>false</useCompoundFile> <mergeFactor>10</mergeFactor> <ramBufferSizeMB>32</ramBufferSizeMB> <maxFieldLength>10000</maxFieldLength> <writeLockTimeout>1000</writeLockTimeout> <commitLockTimeout>10000</commitLockTimeout> <lockType>native</lockType> </indexDefaults> <mainIndex> <useCompoundFile>false</useCompoundFile> <ramBufferSizeMB>32</ramBufferSizeMB> <mergeFactor>10</mergeFactor> <unlockOnStartup>false</unlockOnStartup> <reopenReaders>true</reopenReaders> <infoStream file="INFOSTREAM.txt">false</infoStream> </mainIndex> <updateHandler class="solr.DirectUpdateHandler2" /> <requestDispatcher handleSelect="true" > <requestParsers enableRemoteStreaming="false" multipartUploadLimitInKB="2048" /> </requestDispatcher> <requestHandler name="/update" class="solr.XmlUpdateRequestHandler" /> <requestHandler name="/admin/" class="org.apache.solr.handler.admin.AdminHandlers" /> <requestHandler name="/update/csv" class="solr.CSVRequestHandler" startup="lazy" /> <requestHandler name="/update/json" class="solr.JsonUpdateRequestHandler" startup="lazy" /> <requestHandler name="standard" class="solr.SearchHandler" default="true"> <lst name="defaults"> <str name="echoParams">explicit</str> <str name="defType">edismax</str> <str name="qf"> name^1.8 locid^2.0 state^1.0 </str> <str name="q.alt">*:*</str> </lst> </requestHandler> <admin> <defaultQuery>solr</defaultQuery> </admin> </config> # /tmp/playground/data/weather/conf/schema.xml <?xml version="1.0" ?> <schema name="weather lookup index" version="1.3"> <types> <fieldType name="integer" class="solr.TrieIntField" precisionStep="0" omitNorms="true" positionIncrementGap="0"/> <fieldtype name="string" class="solr.StrField" sortMissingLast="true" omitNorms="true"/> <fieldType name="location" class="solr.LatLonType" subFieldSuffix="_coordinate"/> <fieldType name="tdouble" class="solr.TrieDoubleField" precisionStep="8" omitNorms="true" positionIncrementGap="0"/> <fieldType name="text" class="solr.TextField" positionIncrementGap="100" autoGeneratePhraseQueries="true"> <analyzer type="index"> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/> <filter class="solr.LowerCaseFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/> <filter class="solr.LowerCaseFilterFactory"/> </analyzer> </fieldType> </types> <fields> <field name="name" type="text" indexed="true" stored="true" multiValued="false" required="true"/> <field name="state" type="text" indexed="true" stored="true" multiValued="false" /> <field name="locid" type="text" indexed="true" stored="true" multiValued="false" /> <field name="loc" type="string" indexed="false" stored="true" multiValued="false" /> <field name="type" type="string" indexed="false" stored="true" multiValued="false" /> <field name="rank" type="integer" indexed="true" stored="true" multiValued="false" /> <field name="geoloc" type="location" indexed="true" stored="true"/> <dynamicField name="*_coordinate" type="tdouble" indexed="true" stored="false"/> </fields> <defaultSearchField>name</defaultSearchField> <solrQueryParser defaultOperator="OR"/> </schema>

Now that its installed and running lets test it.

$ cd /tmp/playground $ java -Dsolr.solr.home=data -jar start.jar

Now Fire up your favorite browser i.e. Chrome just a suggestion ;). Go to http://localhost:8983/solr this is the admin interface to the solr service. Now that Solr is running we can load data into lucene. It is easy as pie, we can just import a csv file to be indexed. So create the file /tmp/city.csv see below. Next use curl to POST the city.csv to the RESTFul Solr service to have it load/index that file. This loads the cities  and latitude, longitude, zipcodes, etc.

# /tmp/city.csv name,state,locid,loc,type,rank,geoloc "woodland park",NJ,,"40.8892,-74.195",PLACE,4,"40.8892,-74.195" Denver,NY,12421,"42.27,-74.53",ZIP,2,"42.27,-74.53" Denver,PA,17517,"40.24,-76.13",ZIP,2,"40.24,-76.13" Denver,NC,28037,"35.51,-81.01",ZIP,2,"35.51,-81.01" Denver,IN,46926,"40.89,-86.05",ZIP,2,"40.89,-86.05" Denver,IA,50622,"42.66,-92.33",ZIP,2,"42.66,-92.33" Denver,MO,64441,"40.42,-94.29",ZIP,2,"40.42,-94.29" "Denver City",TX,79323,"32.96,-102.82",ZIP,2,"32.96,-102.82" Denver,CO,80201,"39.69,-105.08",ZIP,2,"39.69,-105.08" Denver,CO,80202,"39.74,-104.99",ZIP,2,"39.74,-104.99" "denver, centennial airport",CO,KAPA,"39.56389,-104.8483",ICAO,9,"39.56389,-104.8483" "denver, denver international airport",CO,KDEN,"39.83278,-104.6575",ICAO,9,"39.83278,-104.6575" "denver / stapleton international, co.",CO,KDNR,"39.78333,-104.8666",ICAO,9,"39.78333,-104.8666" "denver nexrad",CO,KFTG,"39.78333,-104.55",ICAO,9,"39.78333,-104.55" DENVER/ARTCC,CO,KZDV,"40.1833333333333,-105.133333333333",ICAO,9,"40.1833333333333,-105.133333333333" "DENVER CITY 7W",TX,XDVS,"32.9833333333333,-102.933333333333",ICAO,9,"32.9833333333333,-102.933333333333" "North Douglas County Below 6000 Feet/Denver/West Adams and Arapahoe Counties/East Broomfield County",CO,COZ040,"39.72,-104.80",ZONE,10,"39.72,-104.80" "Denver International",CO,KDEN,"39.8333,-104.65",AIRPORT,5,"39.8333,-104.65" "DENVER F. RANGE",CO,KFTG,"39.7833,-104.55",AIRPORT,5,"39.7833,-104.55" DENVER/ARTCC,CO,KBJC,"40.1833,-105.133",AIRPORT,5,"40.1833,-105.133" "DENVER WATER DEPARTMENT",CO,KBKF,"39.729,-105.009",AIRPORT,5,"39.729,-105.009" "DENVER HEALTH",CO,KBKF,"39.727,-104.991",AIRPORT,5,"39.727,-104.991" "DENVER FEDERAL CENTER HELISTOP",CO,KBJC,"39.723,-105.111",AIRPORT,5,"39.723,-105.111" "DENVER POLICE DEPARTMENT-DISTRICT 3",CO,KAPA,"39.687,-104.96",AIRPORT,5,"39.687,-104.96" "DENVER ARTCC",CO,KBJC,"40.187,-105.127",AIRPORT,5,"40.187,-105.127" "DENVER OF THE EAST",NC,KIPJ,"35.493,-80.966",AIRPORT,5,"35.493,-80.966" "DENVER CITY",TX,KGNC,"32.975,-102.842",AIRPORT,5,"32.975,-102.842" "denver county",CO,,"39.76,-104.83",PLACE,4,"39.76,-104.83" "denver nexrad",CO,,"39.78333,-104.55",PLACE,4,"39.78333,-104.55" denver,CO,,"39.76800,-104.87270",PLACE,4,"39.76800,-104.87270" denver,IA,,"42.67120,-92.33410",PLACE,4,"42.67120,-92.33410" denver,IN,,"40.86420,-86.07640",PLACE,4,"40.86420,-86.07640" denver,KY,,"37.77590,-82.85510",PLACE,4,"37.77590,-82.85510" denver,MO,,"40.39900,-94.32370",PLACE,4,"40.39900,-94.32370" denver,NC,,"35.53110,-81.03000",PLACE,4,"35.53110,-81.03000" denver,NY,,"42.2125,-74.56972",PLACE,4,"42.2125,-74.56972" denver,PA,,"40.23250,-76.13870",PLACE,4,"40.23250,-76.13870" "denver city",TX,,"32.96960,-102.83070",PLACE,4,"32.96960,-102.83070" Denver,IL,,"40.28330,-91.10000",PLACE,4,"40.28330,-91.10000" Denver,NE,,"40.434655,-98.66677",PLACE,4,"40.434655,-98.66677" Denver,TN,,"36.04694,-87.92083",PLACE,4,"36.04694,-87.92083"

$ cd /tmp/ $ curl http://localhost:8983/solr/weather/update/csv?commit=true --data-binary @city.csv -H 'Content-type:text/plain; charset=utf-8'

Next lets test the search by trying a few search results out.

http://localhost:8983/solr/weather/select?q=Denver

This will find all matches that have Denver. Our solrconfig.xml also lets Solr know that we want to boost fields using “name^1.8 locid^2.0 state^1.0” So the name field is more important than the state, but the zip code trumps all. For example if you had a city named “cool town” and a state “co” the “cool town” would be the first match then the state. Now lets try:

http://localhost:8983/solr/weather/select?q=Denver&sfield=geoloc&pt=40.08,-105.36&sort=score%20desc&bf=recip(geodist(),2,200,20)

This kicks in the Geo spatial magic, look at the fields sfield, pt, sort, bf fields. The sfield specifies which field is used to calculate the distance from pt. The pt is the point that all results will be measure against. In this example, (40.08,-105.36) is Boulder, CO next to Denver. The bf lets Solr know to boost the search results using the calculated distance from (40.08,-105.36) so Denver, CO wins over Denver, TX because Denver, CO is closer to Boulder, CO.

Javascript is such a cool language because it is so flexible. A friend challenged me to get JavaScript to support a feature that is commonly used in C#. In C# you can have custom attributes that can be applied to class, functions, arguments, packages, etc. This is very useful if you want to add metadata or declarative information to the code. I have built many things using this feature in C# like a REST API that uses the attributes to define documentation, ACLs, field validation, etc. This is very powerful because the code is very clean and declarative.

For exampl, lets say you want to make a generic REST API library so people can easily expose functions to be called throw a RESTful API. The standard way of doing this is to require the developer to explicitly add each function they want exposed to the REST interface by defining a configuration that maps a function to a RESTful endpoint. This setup and configuration code is separate from the code that has all the logic. Alternatively if you wanted a way to add metadata to the code that would not effect the way the function executes and stores the configuration details you could comment the code and use reflection to retrieve the metadata. C# and Java support this feature as a first class citizen but I do not know of a way to do this in JavaScript. So how can we do this? 

Let's enter into the world of JavaScript and why I love it! JavaScript is interpreted at runtime so we can just ask for the code and parse out the metadata. So how do we do this its easy! I will walk you through the code below.

function parseMetaTags(metadata) { var tags = metadata.match(/\[@(.*?)\]/g) , restult = {}; if(!tags) { return false; } for(var i in tags) { var str = tags[i] , len = str.indexOf('{') , tag = str.substring(str.indexOf('@') + 1, (len != -1 ? len : str.length - 1)) , details = (len == -1 ? true : eval('(' + str.substring(len, str.length - 1) + ')')); restult[tag] = details; } return restult; } Function.prototype.reflect = exports.reflect = function(obj) { obj = obj || this; if(typeof obj === 'function') { if(obj.args) { return false; } var fnStr = obj.toString().replace(/[\n\t]/g,' ') , argStr = fnStr.match(/\(.+?\)/) , len = (argStr ? (argStr.index + argStr[0].length) : 11) , fnBodyStr = fnStr.substring(len) , metaData = {func:[], args:{}}; if(argStr) { argStr = fnStr.substring(argStr.index + 1, len - 1); var argList = argStr.match( /(([$\s\w]+)?(\/\*.+?\*\/)?([$\s\w]+)?)+/g); for(var i in argList) { if(argList[i] != '') { var meta = argList[i].match( /(\/\*.+?\*\/)|([a-zA-Z_$][0-9a-zA-Z_$]*)/g); var key = false , val = []; for(var j in meta) { if(meta[j][0] == '/') { var details = parseMetaTags(meta[j]); if(details) { val.push(details); } } else key = meta[j]; } metaData.args[key] = val; } } } if(fnBodyStr) { var bodyList = fnBodyStr.match(/(\/\*.+?\*\/)/g); for(var i in bodyList) { if(bodyList[i] != '') { var details = parseMetaTags(bodyList[i]); if(details) { metaData.func.push(details); } } } } obj.args = metaData.args; obj.func = metaData.func; return true; } else if(typeof obj === 'object') { for(var i in obj) { exports.reflect(obj[i]); } return true; } return false; } /* #### EXAMPLE #### */ function exampleAddUserToDB( arg1 /* [@REST{postName:'name', type:'string'}] */ , arg2 /* [@REST{postName:'age', min:16, max:100}] */ ) { /* [@REST{type:'POST', access:'public'}] */ return true; } exampleAddUserToDB.reflect(); console.log(exampleAddUserToDB.args); console.log(exampleAddUserToDB.func);

So what is going on in the code is very basic. The idea is we want to look for comments in the code that are formatted like /* [@attribute{json}]*/ and parse the comments out and turn them into metadata that can be attached to the function or its arguments. Because we want to add reflection to everything we add a prototype to Function. This makes it possible for any function to have reflect capabilities. The reflect function will call the toString() on itself. This gets the string representation of the function including all the comments so its just a matter of parsing out the tags in the code. After getting all the tags, we add the parsed tags onto the function itself so its easy to ask the function for its metadata. The function that parses the tags is parseMetaTags so you could build any kind of parser that would return any type of tags i.e. string, function, object, numbers. 

This lets you add any kind of metadata to your code via comments that follow a standard format. Than at any time, you can call reflect that will parse out all the metadata and return an object that your code can query against. Lets take an example of having a RESTFul service. We now can comment in our code to let the RESTFul controller query to find out if the function should be called using GET, POST, what the endpoint URL is, rules around required arguments, inline documentation that the RESTFul service can return, etc. Basicly anything you can dream up. When you call the exampleAddUserToDB.reflect() it parses out the metadata so it can be accessed by exampleAddUserToDB.args and exampleAddUserToDB.reflect.func. Below is the output of the example above. As you can see all the metadata has been pulled out of the code. You can now build any function you want and add metadata to it and use reflection to retrieve it. What is more amazing is this is not apart of the JavaScript language but we still can support it in just a few lines. That is what I love about JavaScript it is so powerful.

{ arg1: [ { REST: { postName: 'name', type: 'string' } } ], arg2: [ { REST: { postName: 'age', min: 16, max: 100 } } ] } [ { REST: { type: 'POST', access: 'public' } } ]

I am not saying this is a good way to solve a problem but it is cool that we can do stuff like this. I love many things about C#, ObjC and Java each have their strengths and weaknesses but JavaScript seems to be more terse and flexible. This is both a good and bad thing! I really thing JavaScript is very very powerful, exciting, and fun to use and I hope sharing my weekend playground. I hope this with inspire you and your creativity to do other crazy cool things with JavaScript that it was not designed to do.

This video elegantly explains why it is so important to change the way we incentivize people to do work. I think we live in amazing times  where companies like Google, Facebook, GitHub, Bettermeans, can change the way things are done. They are proving  traditional hierarchical structures cannot create game changing productivity.   Conversely groups of passionate people and have the force of an Army.

Robert Kieffer, an a engineer at Facebook and the company's internal Node.js evangelist, gave a talk at NodeConf today. He says the company isn't doing enough with Node.js yet. So instead of an explanation of how to deploy Node.js to thousands of servers, he's talked about why Facebook needs Node.js and the things holding the company back from adopting it widely.

You can find slides from this presentation here (PDF).

Some critics aren't impressed because many of the problems Node solves can be solved with other technologies such as libev, Tornado, EventMachine, etc. So why should we be interested in Node.js? Kieffer because it breaks down the silos between developers. He says that at almost every place he's worked, people are pigeon-holed into specific roles: front-end developer, backed-end devloper, etc. But that means that it's hard for developers to pitch in to help other teams.

For example, Kieffer once worked for a company that was making an online document editor, back before Google Docs. He was the only front-end JavaScript developer on the project. A customer asked the company to build a Web-based calendar mockup. The CEO of the company, eager to please, decided that the team should actually build a full-featured app, not just a mock-up. Kieffer was stuck building the whole thing in a very short period of time by himself, because the other developers on the team weren't able to write JavaScript. In a world where the same language was used for front, middle and back-end development, more people could have pitched in on the project.

So why isn't Facebook widely embracing Node.js yet? Facebook prioritizes performance, as illustrated by HipHop. Even 1% improvement matters. But reliability is just as valued. Facebook has 600 million users - they will find any weakness in any new technology introduced. For a company like Facebook, there's no one for its developers to look to as an example for how to do things. Also, rolling out new technology to thousands of servers is a non-trivial matter.

However, Facebook is using Node for a few things. One is JSGameBench, an HTML5 game benchmarker. Another is a mobile JS framework that has yet to be announced, but will probably be open-sourced. But the bulk of the Node.js work is being done by the Chat team for traffic analysis and load testing.

If your like me, you just want to know how something works and some basic understanding of the motivation behind the decisions that went into the design of the software. I could not find a good introduction to Git written this way so I will try. I am not going to walk through how to install Git you can find good instructions on the Internet or just download it here.

Git is not like CVS, Subversion, Pee, Bazaar, etc. It was trying to solve the problem of being a distributed version control system so there was no need to have a centralized server. The way Junio Hamano and Linus Torvalds solved this problem was to create a database that stores all the commits as atomic files with an acyclic graph that holds the commit states. I will describe this in detail later but for now the key point is that Git does not store individual changes to files or deltas like traditional version control systems.

Traditional systems store information as delta points. For example, file A has two delta points one at version 2 and another at version 4. Each delta stores how the file was changed like any new, deleted, or modified lines not the entire file. If you want the file A version 5, you have to replay the two deltas in order. Making each change stored in the deltas at version 2 then version 4.

Git doesn’t think of or store its data this way. Instead, Git thinks of its data more like a set of snapshots of a mini file system. Every time you commit, or save the state of your project in Git, it basically takes a picture of what all your files look like at that moment and stores a reference to that snapshot. To be efficient, if files have not changed, Git doesn’t store the file again—just a link to the previous identical file it has already stored. Git thinks about its data more like the Figure below.

So all the nodes with dotted borders are not stored but are links that point to the contents of the file that has not changed. For example, A1 with the dotted border points to the node labeled A1 with the solid border. All the green nodes with a solid border are stored in Git repo. Each version is just a list of pointers which we will explain later.

Each repository has a full copy of this database and graph that is why Git is so fast. When you create a new repo or clone a repo it creates a hidden directory named .git that contains the complete database and all supporting files like configuration. Lets start by creating an empty repo and add a file.

Now its important to talk about how Git stores its versions. All the magic is in the .git/objects directory. You can think of this directory as a key-value store where the key is a SHA-1 hash of the value and the value is a file. When you commit, or save the state Git will hash all the files it is tracking and check if it needs to add them to the key-value store. So the store will always have a complete copy of the original file and you can always look it up with its hash code.

For example, in the scenario above when the file A is committed to version 1 Git hashs the file A and check if it is in the store. In this case, its new so it adds it to the store. In the next commit file A is modified and committed to version 2 so file A’s hash is different so it needs to be added to the store because the hash code is new. In version 3 file, A is not modified so its hash is the same as version 2’s hash and Git does not need to store the file because Git already has a copy of that hash.

If you look at the directory, you will also see that the SHA-1 hash are partition in subdirectories. If you take the directory name and the file name an join them you will have the full hash. The files are compressed so you can not just view them but you can use the command below to view the files contents.

$ git cat-file -p {sha-1-hash}

Now that we understand how Git stores all its files lets talk about how it tracks versions. As you can see everything in git is referenced by a SHA-1 hash so if you want to build an acyclic graph that represents a sequence of commits its easy. Just look at the Figure below.

As you can see each commit points to a tree that has a list of pointers to the files Git stored in its key-value store. The tree node holds additional meta data because Git stores the files contents not its path and meta data so we have to save this information elsewhere. If your observant you will notice that the commit and tree nodes link to each other using hash code. This is because Git stores the version graph the same way it does for files. So each node in the graph are just files and each file/node can reference other nodes so all you need to start walking the graph is a hash code.

When we need to go back to a snapshot we just locate the tree and get a list of all its pointers and start copying the files from the Git store to the disk. Now this comes to another big difference with the way Git works over traditional version control systems. Git needs to understand what it is tracking. Git is just a database with the ability to take snapshots of your work so you need to let it know what to track. Unlike traditional systems if you have modified 5 files in your work area you can tell Git to only track or snapshot 2 of the files in a commit. This is very useful but confusing sometimes, so lets take an example.

If you create an empty directory and add 5 files A, B, C, D, and E and you ask Git that status of the directory you will find Git will list all 5 files as untracked. In-other words you have not told Git that you care about these files and do not want any history tracked. To start tracking a file you need to add it to a snapshot. Once you have added a file to a snapshot Git will start tracking it. If you add A, B, E to a snapshot or add them to a commit Git will start tracking the files. After that if you modify files A and E and ask Git the status it will tell you that out of all the tracked files A, B, E only A and E have changed. It is important to understand the status of tracked files is different then what is in a snapshot. At any point you can ask Git if it sees an differences from its active snapshot and the current directory by comparing hash codes. Git will not automatically add changes into the next snapshot but requires you to explicitly add what is in the next snapshot. So in the previous example even though you edited A and E you can tell Git to only add E's change to the snapshot.

$ cd /tmp/ $ git init my-empty-repo2 $ cd my-empty-repo2 $ touch A B C D E $ git status $ git add A B E # tell Git to add A, B, E to the next snapshot $ git commit -m'version 1' # Git will start tracking A, B, E (its now commit) $ git status $ echo "EDIT A" > A $ echo "EDIT E" > E $ git status $ git status -s # watch the M it is in the second place i.e " M" not "M " $ git add E $ git status -s # now the modified flag is in commit place i.e. "M " not tracked " M" $ git commit -m'version 2' # Git only save E change $ git status

At this point I hope you are seeing how Git is like an acyclic graph and the graph structure is your history. Each snapshot is just a node in the graph. Now this brings us to the real power of Git branching because, your history is just a graph if you want to branch it is just a dag in the graph. You can just create a new node that points to its parent. That is it! Lets briefly talk about sharing your work. Git is just a key-value store with files names with SHA-1 hash codes so if you have two different developers that have forked the same Git repo and they need to merge them its easy. Remember files in Git are just SHA-1 hash and the graph are also stored in SHA-1 so if they are not the same file/node they will have a different hash. So its safe to merge/sync the two Git stores and if you just have a pointer to a node in the graph you can start working the graph. 

I thought it would be nice to start sharing my thoughts on books that i have recently read. Recently I have been reading a lot of business, organization, and marketing books. I just finished reading Tribes, by Seth Godin. I have been noticing a trend toward community and decentralization in business. I believe this is a part of the disruption that many high tech companies are encouraging. Seth Godin, will make you think about the opportunities for leading your fellow employees, customers, and community. Many of the books I have read in the past month have had a significant impact on my views of leadership, and the direction of our industry. I believe tribes have always been a huge influence on the direction of our evolution. Our future has always been a struggle between our ability to innovate with technology and our social nature. For what ever you think, we are a social species, and much of the research is revealing our biology controls and influences many of our decisions. Tribes is very inspiring and reminds me of many other books I have read including lectures I have recently watched like Ori Brafman, and Geoffrey Moore on Stanford's Entrepreneurship and thought leaders. A few thoughts on the book:

“Today, marketing is about engaging with the tribe and delivering products and services with stories that spread.” - Build it and they will come mentality is dead! You need to engage your community and support it because building your channel is no longer an easy as it was. Technology is letting your customer/community subvert your control easier so you need to become a leader. The days of being a dictator are numbered and if you can not control people with power you have to influence them. An another wonderful book is "What Would Google Do?", by Jeff Jarvis. In this book Jeff talks about what google has done to change the game. Because things are more interconnected and we have less control and influence over the market we have to become a better citizen. 

"In order to lead a movement you need two things; and shared interests and a way to communicate." - When you join a tribe, you will find out it has a center of gravity and you want to be closest to that center. This is a place where people of like mind commune and share. Your message must be on topic and compatible to the tribes core dynamics. This is as much as identifying who is not a part of the tribe and who is. The best way to inspire people at the core of the tribe is to have a genuine passion. Communicate using all possible methods and engage your community. Joining a tribe should be a long-term commitment. Leaders naturally find this center and  understand the dynamics of the community to help make small course corrections to influence the group. This is where you want to be and how to lead a movement.

"The status quo is persistent and resistant. It exists because everyone wants it to. Everyone believes that what they have got is probably better than the risk and fear that come with change." - Tribes are changing this! When people start to connect and share the influence they have on the environment around them is unstoppable. Today people are motivated, and connecting with each other faster than ever before. The status quo is just society fancy. As a tribe and leader, you must persuade people to join your tribe. Most people want to be part of the healthiest and strongest community so any effort made to empower your community ensures growth. Do what you love, follow your passion, shares your story, and people of like mind will join you.

"If your organization requires success before commitment, it will never have either." - Enough said 

"People don’t believe what you tell them. They rarely believe what you show them. They often believe what their friends tell them. The always believe what they tell themselves. What leaders do: they give people stories they can tell themselves." - 

It looks so easy, Just click to add account, right? Nope! When I was at Fuser we spent a few sprints figuring out how to auto detect what kind of account the user was trying to add. For example if you had an account [email protected] is could be a IMAP, yahoo mail plus, or POP3 account. Each time you detected it would take a long time because you would have to try many different ports and protocols. Then you would need to pick the best one i.e. IMAP is better then POP3 if the mail server supports both. I also remember all the hard work that went into the UI and trying to handle all the async callback handling for the applet that had many threads running at the same time. Caching the domain so we could look up to see if we could skip the discovery phase. Oh, I miss the fun of working on a website that had a Java applet, .Net back end, and a very complex UI running in javascript. Having to work in C#, Java, and javascript and getting it all to work! Smial!!!

To enable social login and registration you need to have a free, pro, or enterprise account with Janrain and setup your providers i.e. Facebook, Twitter, etc. Next you need to provide a callback endpoint that Janrain can POST back to. Once Janrain POSTS to your site you can use the Janrain API to validate the credentials and request a normalized version of their profile, contact list, etc.

You can signup for a free, pro, enterprise account at Janrain.

The signup process uses their social login so you will need to have an account with Facebook, Google, Twitter, etc to start. You will need to confirm your information and create your account. You will need to pick your application's name. This name is visible by your users and will be presented to them each time they grant access to their 3rd party account i.e. Facebook and Twitter. Check out the image on the right hand side to see what your application name will look like when your users try to connect.

Great! You should be at your dashboard and if not go to your dashboard. You can keep clicking the button in the upper right hand corner labeled "account" or "dashboard" and this will toggle you between your account and dashboard--- how clever. Step 1 is to click on the "Sign-In for Websites" under the "Quick Links" on the right-hand side. This will take you to the wizard to help you get the javascript code to put on your site.

Before you do that, you need to choose and configure the providers that you want to offer to your users. To do this, skip the "Get the Widget" step by clicking on the "Choose Providers" this is labeled as Step 2. If you pick providers that require additional configuration i.e. ones that have a gear next to them you will be prompted to configure them.

Step 3 is to configure the providers. You will need to follow the steps in the wizard and copy the key/secret from the providers into Janrain setup wizard. Above is an example of Facebook and Twitter providers. Janrain will help you configure all your providers if you have a pro or enterprise account.

Now that you have picked your providers and configured them click on the wizard step labeled "Handle Tokens" copy the "apiKey" value this is your api key and SHOULD be kept secret. You can also get this value on your main dashboard page. We will need this later when we write our server code so keep it handy. 

If you want to test your configuration you can optionally click on the "Test Tool" under the "Resources" on the right hand side. This will simulate the Janrain sign-in widget and show you what data is returned when someone connects to your site.

We are almost done. The final step is to go to your dashboard and click on "Settings" under the "Quick Links" on the right-hand side. If you plan to use Janrain on a domain other than localhost you will need to add that domain to the "Token URL Domains". There are also many different settings like "Provider Configuration" on the dashboard page. I would recommend exploring them later after you get your code up and running.

Let the fun begin, now install the Janrain node.js package you can do this using npm or visit https://github.com/demetriusj/janrain to manually install the package. We are also going to need to build a barebone site to demo all our cool work so lets use connect because it makes it so much easier. If you do not have connect installed use "npm install connect"

$ npm install janrain-api

Now that you have the package installed you will need to create a folder for the project. We need to make a host demo site and RPX callback hook. This is just a place Janrain can POST back to.

Now create a file app.js in the folder. You will need to update the {YOUR_API_KEY}, {YOUR_APP_NAME}, and optionally localhost:8080 if you are going to run the application on a different domain than localhost. The {YOUR_API_KEY} should be replaced with your api key found in your Janrain account. The {YOUR_APP_NAME} is the name you gave your Janrain account.

var connect = require('connect'), janrain = require('janrain-api'), express = require('express'); var engageAPI = janrain('{YOUR_API_KEY}'); var app = express.createServer(); app.get('/', function(req, res){ res.render('index.jade', { locals: { app_name:'{YOUR_APP_NAME}', domain:'localhost:8080' }, layout:false }); }); app.post('/rpx', connect.bodyDecoder(), function(req, res){ // We will cover this in the next session! }); app.listen(8080);

Create a view lets use jade because that is what all the cool kids are using. Lets call the file views/index.jade and copy the code below.

!!! 5 html(lang="en") head title Janrain Web app :javascript | var rpxJsHost = (("https:" == document.location.protocol) ? "https://" : "http://static."); | document.write(unescape("%3Cscript src='" + rpxJsHost + "rpxnow.com/js/lib/rpx.js' type='text/javascript'%3E%3C/script%3E")); :javascript | RPXNOW.overlay = true; | RPXNOW.language_preference = 'en'; body a.rpxnow(onclick='return false;', href="https://" + locals.app_name + ".rpxnow.com/openid/v2/signin?token_url=http%3A%2F%2F" + locals.host + "%2Frpx") login

Now lets fire up the demo.

$ node app.js

If you open a browser up and go to http://localhost:8080 you should see a link "login" and if you click on it you should see the Janrain popup. We are almost done and all we need to do is wire up the callback so Janrain can pass us the identiy and we can validate it and sign in the user. To do this we need to write the RPX callback and we will do that next. Just stop node running and edit the app.js file and replace the app.post with the code below.

app.post('/rpx', connect.bodyDecoder(), function(req, res){ // STEP1: Get the token sent by the widget // and validate it. var token = req.body.token; if(!token || token.length != 40 ) { res.send('Bad Token!'); return; } // STEP2: get the auth info using the Janrain API. // Janrain will validate the request and return the profile // details using a secure channel. engageAPI.authInfo(token, true, function(err, data) { if(err) { res.send(err.message + ( data ? (' -- ' + JSON.stringify(data)) : '')); return; } /* check if we assigned a unique identifier to this user and if so they user is a return user if not we need to create a account in our user management system. You can also make other API calls to get the user contact list, friends, etc. if(data.profile.primaryKey) { signInUser(data.profile.primaryKey); } else { var uid = create a new user in your management system. engageAPI.map(data.profile.identifier, uid, function(err, data) { if(err) {res.send(err.message); return;} signInUser(uid); }); } */ res.send(JSON.stringify(data)); }); });

Just write your own signInUser function. This function will login the user into your site. This can be done using cookies or a variety of techniques. Notice that we also have sample code that lets you map the user to your user management system. So next time Janrain identifies the user it will identify them with your unique ID not the provider's ID. Its that simple now you can let your user sign in and register with any provider. Check out all the data that comes back from the authInfo call at Janrain API Docs. 

If you look at the view you will see I am manually creating the href that starts the Janrain popup. The rpx.js script helps wire-up all the magical UI. It looks for an href on the page with a css class "rpxnow"and uses the href to tell the widget what Janrain account to link to and what is the callback url. Looking at the url you will see the format is {application name}.rpxnow.com/openid/v2/signin and has a query string "token_url" that tells the widget where to POST the callback hook. If you remember in the Janrain wizard where we setup the providers their was a tab for "Get the Widget" on this tab it create sample code that you can cut/copy/paste but here in the example we just hide it into the template.

I recently had the pleasure to work on Janrain for one of my projects. So what is Janrain? They are leaders in the social login/registration and sharing space. They're one of the most inexpensive solutions to provide a turnkey social login and social sharing solution. We evaluated Gigya and the work required to write it from scratch and picked Janrain. Janrain Engage lets a website’s visitors register or login with their existing social network accounts from Facebook, Google, Twitter, Yahoo, LinkedIn or other networks, and share content or their site activities with friends on multiple social networks. Additionally Janrain lets users import their social network profile data and invites their friends to visit your website. They offer other services like analytics, profile storage, badges and points. They have been around for a long time and started OpenID.

The main benefit’s that I see to using social login is that we get the users social profile and social graph. We can provide a more customized experience and target advertisements more effectively. Finally, we can tap into the viral social web and drive traffic to our sites. From a user standpoint, they only have to manage one account to identify themselves and their experience on the site is more social. So its a win win for both sides. 

Key differences between using Janrain or Gigya over developing against native APIs like Facebook's connect are that you do not have to write lots of code to support all the different providers. Both Janrain and Gigya have support for dozens out of the box. You also do not have to maintain and keep up with all the changes to the API, and figuring out how to normalize the data. For $10 a month or $100 a year that is worth it. You can even start with a free account with limited features or get an evaluation account for a limited time. I know the world is trending towards using Facebook and Twitter widgets but I like the idea of supporting many different providers. I will say many of the provider widgets are amazing, and I think there is a place for Janrain and native widgets on the same site.

So how does it work?

Its simple you provide a login button that fires up the Janrain widget (just a little script you need to include on your page). The user then can click on one of the providers like Facebook, Google, Twitter, or other providers to identify themselves to your site. When they click on which provider they want to be identified with they will be prompted by the provider asking if the provider can share your identity so that it can be used by your site. For example, let’s say a user logs-in using Facebook they would be prompted to connect to your site with details about what is going on. If the user is already signed in to Facebook, they will be asked if its OK to share certain information with your site. The degree of information your site is asking the user to share is controlled by your configuration inside Janrain. So you can ask the user to share as little information about them self like their full name to detailed information including all their friends, contacts, DOB, etc.

Janrain also tries to simplify and consolidate the login process because each provider has a slightly different UI and flow. They also try to streamline the work flow so if the user is a return user they do not have to remember which provider they connected with. All these small details make the process a delight for your users.

Once the user has connected and shared their profile data with your site you no longer need to have your own registration process. The provider will share the details about the user so you can skip asking for their email, name, etc. and just create the account. The user will not need to have their own username and password but, they can be simply identified by the provider. Optionally you can store additional properties and preferences that are for your site only but link to the providers profile. At any point, you can ask the provider to share the most up-to-date information on that profiles social graph, contacts, etc.

Janrain is not a replacement for your user management and authentication system. Your site will still need to keep track of your users but, the main difference is that it does this by a user’s identity not an username and password. Janrain will let you associate a unique identifier to any providers identity. For example, if a user identifies themselves through Facebook you can tell Janrain next time the user signs back into your site to identify them with your previously assigned unique identifier, not with the default provider's identifier. Simplifying the process of having to map Janrain ID to your user IDs. Furthermore, you can tell Janrain to identify multiple profiles to a single ID, this is useful if a user has a Facebook and Twitter profile but you want to identify them as the same user.