MoneyDesktop Tech

Web application development has been rough river of continuous evolution.  We have come a long way from the days of mostly static html content. Today there is such a variety of tools that we can use to bring our wildest imaginations to life.  It is becoming quite common now to leverage Javascript frameworks such as backbone or spine with our json formatted Rails API's to create a seamless user experience that comes relatively close to feeling like a native application.  Another step in this evolution is creating real time event driven sites with a pub/sub server like Faye.  This will allow you to cut out all of those polling requests that are constantly pounding your web server for the very latest information and replace them with a more event driven style experience.

In the coming article we'll go over setting up a Faye server inside a standard Rails app.  Additionally we will cover some basic security to allow for private channels and how you can add custom extensions to your server.  Finally some suggestions about how to implement the client side Javascript and hopefully give you a bit of inspiration and a few ideas to up the game on your latest web app creations.

Setting Up Your Server

Faye lets your server side Ruby communicate with your client side so easily that you'll ask yourself, is that really it?  Ryan Bates from RailsCasts did a wonderful job in episode #260 introducing Faye and then he followed it up with a private_pub gem to facilitate private channels.  This is a valuable resource along with the Faye site itself (http://faye.jcoglan.com/) which has some great documentation for setting up your server in both Ruby and Node.

Getting your Faye server going is as easy as adding 'faye' to your Gemfile and then just a few lines in a rackup file. See Figure 1 for a sample faye.ru.

https://gist.github.com/3343400

Notice the log_level set to debug.  This is helpful when getting started so you can see the network chatter that happens as you interact with your server.  As you get to your production environment you would of course want to turn this off to help with performance and to prevent bloating your log folder.  Using a yml for configuration helps with setting the domain, port and later the ssl information for the various environments. 

Faye runs on one of several different event-driven servers.  Check the site for the latest information.  Thin was an easy choice and seems to work quite well.

If scaling is an immediate concern you may want to consider setting your "engine" parameter of your server configuration.  One of the supported engines is Redis, by setting this up you will be able to have multiple servers and it wont be a problem.  Again using a yml file for your redis configuration your rackup file may now look something like this.  Figure 2

https://gist.github.com/3343470

Client Side

One of the things I love about the Ruby Faye server is that it delivers up its own JS library that you can include that will handle all the heavy lifting in communicating to your server.  All you need to do is include the js file found at the root of your domain that you set the server to run at, for example:

https://gist.github.com/3343687 

If you are unfamiliar with some of the awesome Javascript frameworks out there like backbone.js or spine I would highly recommend doing some research into what these can do to improve your existing JS code base. Many Rails developers take for granted the convenience of Active Record as the ubiquitous ORM layer.  Adding a similar sense of organization to your client side model layer can bring a lot of power to your web app.

In your JS simply add a connector and an event listener and you are off and running. If you're using something like backbone in your application to store your data models you can enhance your event handler to load in the json passed up from your ruby server which will leverage all the magic in backbone to keep your views up to date.  Package your published event in an event wrapper that has a object_type, event_type and a payload.  The event type could be something along the lines of 'created', 'updated', 'deleted'.  You could then grab your model in your backbone store by the id and perform the appropriate action.  Here is a sample JS connector and event listener methods

https://gist.github.com/3343585

You could also take the route of having different listeners for different channels.  Maybe you have a create and update channel for a specific model type by connecting to a very RESTful looking URI scheme.  This would also work but you may end up with some more complex looking JS to handle all of the various channels. 

Security

If your application has private user information that you'd like to only publish to specific clients this is also easy to handle with Faye.  Out of the box you are able to subscribe to different channels just by connecting to different URI's on your server, /events versus /my_users_events for example.  However what would prevent someone from tinkering with their own socket connections and listening in on someone else's communication?

Ryan Bate's private_pub gem was written to handle some of this for you.  However it is quite painless to handle this on your own.  It also removes some of the magic of what is going on under the hood.  The Faye server has an interface for adding extensions that makes it easy to plug in a custom authentication classes and other goodies. When an incoming message is received it is possible to inspect the message type and handle it on an individual level.  For example if you wanted to check all channel subscription requests and verify that the client has access to the appropriate channel simply creating a class like in Figure 4 and then adding it as an extension in your rackup file would give you the ability to have private channels to publish your events on.

https://gist.github.com/3343500

If connecting to your server from Javascript, simply pass the timestamp and access key on initialization for the round trip verification.  You will need to modify your connection method to add the additional authentication information for the connection event.  The Javascript interface is similar to Ruby in that you add an extension to your Faye client, Figure 5 has an example.

https://gist.github.com/3343626

For SSL support you can also configure your key and certificate information into the rackup file.  These are additional parameters on the server listen command.  The final look of your faye.ru may be something like Figure 6. Notice the addition of the security extension mentioned above.

https://gist.github.com/3343489

Conclusion

By adding a simple pub/sub server to your Rails apps you can very quickly create a real time experience for your users without having all the network overhead of polling.  With this guide and all of the many resources online you should have a clear path to getting setup and running with your own realtime environment with Faye.  Combined with the power of a JS framework like backbone your client side views could instantly display new data as it receives updates.

- Bart Elison

Team Lead, MD Dev

Drinking beer doesn't make you fat, it makes you lean . . . Against bars, tables, chairs, and poles.

Join us for our first hosted meet-up at Squatters Restaurant. Come learn about the MoneyDesktop front-end, back-end and mobile teams in a fun and casual setting. Tickets and shirts are limited, so RSVP early. FREE food and drinks will be provided. 

We look forward to meeting you! 

The NoSQL storm is in full swing.  The new hotness is getting the flexibility and speed of a document type store.  There certainly are advantages to this type of data storage, but here is an alternative that has a purpose.

In ActiveRecord you have the option to serialize a field.  The huge downside to this is the difficulty in querying for a specific value in the field.  You could use regex or a LIKE type query but the performance hit is significant.  The Postgres hstore extension provides the answer to those issues.  The syntax looks pretty much like a ruby hash, and you can query specifically for the key or value. 

Thanks to the activerecord-postgres-hstore gem you get some added goodness of migration and query support.  He has also gone to the effort of running some benchmarks to compare the speed differences between the hstore and a serialized field.

Here is a real world example of how this could come in handy.  Lets say you would love to treat your model as a classic polymorphic object.  Activerecord gives you a great foundation for that with single table inheritance.  The problem is the child classes you create don’t have the flexibility to have their own attributes.  Using hstore you could define all of the standard shared attributes like normal on your primary model class but if you have one off needs in the children they can be supported too.  

Example:

https://gist.github.com/1637363

Next time you’re considering a NoSQL option but are worried about losing your join syntax and other relational benefits, consider the hstore and the alternative it provides. Love to hear your successes or failures in using hstore or NoSQL collections.

You remember that show V? You know, the one that was a remake of an old ‘83 series; aired for one and a half seasons before they killed it. Well this post has nothing to do with that show.

In a previous post we talked about some of the things we did to speed up website load time. That post was focused mainly on the client (browser) side of things. In this post, we’ll talk about how we sped up our data retrieval.

As any developer knows, N+1 queries are bad. And our system was guilty. Unacceptable. We had to come up with a solution. Before we talk about how we solved this problem though, a little background information:

We use Protobuf RPC to share data between our different service applications. We convert pertinent parts of a record to a protobuf message and send it across the wire to the interested endpoint.

When converting an ActiveRecord object to a protobuf object, our system was querying for one record, then querying for records related to that record, then querying for records related to those.

Before this refactor, all of the information needed for the protobuf conversion existed in the records that were being converted, retrievable by querying for that record. Now the only way to get this information was to execute a separate query to retrieve it (no, eager loading would not work).

The solution to this problem was simple; we needed to formulate a query that retrieved all of this data at once (completely possible). The implementation of this solution was not so simple.

After trying several different approaches, it was decided that we’d turn to the Visitor Pattern (I know what you’re thinking, “if you think you need the visitor pattern, you don’t”, well, it works for us :). We now have a total of 4 queries which can be reduced to one in a future iteration of refactoring.

Here’s an extensive code example of the visitor pattern in Ruby (you may recognize it from the wikipedia article linked above):

First we’ll define the visitable module. This module will be included in the various classes that need an ‘accept’ method.

https://gist.github.com/1635921

Next we define visitable classes. Notice that the CarElement class here is pretty much useless. All it does is ensures that any classes that inherit from it implement the ‘accept’ method. Otherwise an error will be raised. It’s there only because of the Interface that was defined in the wikipedia example (this code is a fairly direct port of that Java code).

https://gist.github.com/1635930

The next piece of this visitor puzzle is what I like to call the “manager” class. It keeps track of what elements need to be visited and ensures that they are, in fact, visited.

https://gist.github.com/1635981

Now that we have all of our visitor pieces in place. Let’s see what an implementation looks like. Notice that nothing is ‘visited’ until the accept method is called

https://gist.github.com/1635999

If you’re not familiar with the visitor pattern, you may need to go over the code a few times to really understand it.

Here’s a gist of everything in a single file

You can use this example to get a feel for how we solved our problem. Pay particular attention to the way the Wheel, Body, and Engine classes are added to an array which is then traversed. Try to think about those classes as ActiveRecord::Relation objects instead.

As software engineers we have to make a lot of assumptions about our products.  How quickly will we need to scale?  How will our customers use the product? Which features will or will not be important in the end?  Obviously one of our best tools in trying to adapt to the answers to these questions is the agile development process.

Recently during a sprint planning it was determined that we needed to optimize our load times.  It just simply was taking far too long for the product to load.  There was an original assumption that user by user customization would be a key feature.  That the pieces of the app that would be assembled would need to be flexible.  This lead us to chop the app up into smaller modules that would be requested at run time.  Additionally all of the data the app would need to function properly was loaded individually after startup. This lead to a lot of http requests on initialization - over twenty.  To compound the issue there is a browser limitation that lets you make only so many requests concurrently.

A feature that I never believed would be widely used was the dashboard.  The dashboard attempts to give a quick summary of all of the different views in the product.  My personal prediction was that it was data overload, that no one would want to look at it. It turns out nearly everyone has it activated.  Unfortunately to provide a relevant dashboard all of the shared code and data needs to be loaded to generate the graphs and display all of the desired information.   With all of that in mind we set about optimizing our start up time.   The approach was two fold (actually three but that will have to be a later post).  First to condense all of the swf modules that made sense, the ones that would always be required anyway, into the primary app.  Secondly to bundle up the data that we already had immediately available into the template so there was no need to make additional http requests.  This process was surprisingly painless once we had decided what needed to be done.   Now we all know that everyones favorite browser is IE… which turned out to give the only roadblock to successfully improving our load times.  In the process of cleaning things up one of the pieces that was cleansed was the html template that initializes our app.  We added the important data that was already available to a javascript hash, known as ‘flashvars’, that gets delivered to the primary swf.  For reasons that are still baffling, IE behaves differently.  One option was to embed them three times, once with the swfobject loader js code and two more instances  with the object and embed tags that IE prefers.

https://gist.github.com/1628983

This was just too ugly to stomach.  The work around was to create several javascript “get” methods and use the flash external interface functionality to retrieve our json objects from the home template.  This worked out very well in the end, and it’s very fast.

In the html template:

https://gist.github.com/1629039

In the flex:

https://gist.github.com/1629048

With these changes we cut the number of required http requests down to just a few that retrieve data from our SOA.  The load time to display the entire dashboard dropped from around 30 seconds to between 5 and 7 seconds.  This was a huge gain but we knew we could do more, which leads to the third thing that was hinted about earlier.  Until then, happy coding.

Evented programming is a genre of programming that’s been around for quite some time. It centers on the idea that control flow in a program is determined by real-time events, rather than a top-to-bottom execution approach. In simpler terms, you can think of it as two buddies having a conversation about something that will happen later:

> Jeff: “Hey, man. Whenever you go home, would you call me so I know?”

> George: “Sure thing, bro.”

George writes down on his to-do list (or tells Siri) that he needs to call Jeff when goes home for the day. When George goes home, he looks at his to-do list and sees “Call Jeff” under the heading “When you go home”, so he calls Jeff to let him know. All this time, Jeff has been running errands and doing whatever he feels like doing. When he gets the call from George he remembers why he wanted the call in the first place, and then acts on that.

Contrived example? How about a code sample to illustrate.

https://gist.github.com/1508219

Creating a SAX-style parser like the one above is really simple with the eventually gem. eventually allows you to write concise event integrations into your POROs (plain old ruby objects). Simply `include Eventually` into the event emitting object and off you go. Any object can register as a listener for any event, and the source emitter object simply emits when conditions are right for the given event.

eventually was built to mimick the NodeJS EventEmitter API. What makes it different from other EventEmitter ports is it’s handling of the various ruby callback methodologies. Be it block, lambda, proc, or even detached method,eventually handles them all the same.

We’re interested to hear about your experience using eventually or similar technologies. Leave a comment below.

During one of our past sprints here at MD we took on the monumental task of upgrading our flex core from the 3.5 to the 4.0 sdk. First, huge props go out to our flex engineers in getting this done so quickly. There were several challenges we ran into during this process that perhaps others would find interesting. There were also a couple of surprising upsides that we didn’t expect.

A quick qualifier we do a lot with modules and I think many of our issues stem from this. I don’t know how many out there divide up their flex applications the way we do in order to speed up the initial download, but I have a feeling this leads to many of the issues we’ve had when converting.

First the gotchas and challenges.

There have been many discussions on how the changes to stylesheets, most notable the namespacing changes, has caused a lot of pain. We also experienced this as all of our global styles seemed to fail. There seems to be a lot of problems getting our modules to use the global styles. We’ve nearly completely abandoned trying to use embedded stylesheets.

ComboBoxes… They are the most evil component in flex 4. I can’t count the number of times the ComboBox has caused us problems. The symptom is the parent container fails to render followed by general flash player instability. I am not completely sure what is to blame for their failure, my guess is a conflict when MX components are being used in the same parent as the Spark Combo. We finally just did a global code search and removed every single one and replaced it with the DropDownList which has been much more stable.

Event Changes. One of the consistent pain points when converting components to spark is the change to the events being fired. For example, if you want to know about a new child inside of a VBox you watch for a different event than if you’ve “added an element” to a VGroup. This will lead to unexpected UI break downs that our QA team has to be hyper vigilant about finding.

Perhaps it is my imagination but it seems like the incremental build time when saving a change in Flex Builder is a bit slower. I didn’t bother to do time trials, but those few extra mythical milliseconds have caused a bit of internal rage.

The MXML layer of Flex was already quite verbose in its declarations. Spark takes this and pumps in about 30 gallons of anabolic steroids. Now every tiny adjustment requires sometimes multiple lines of MXML. No, I am not bitter about this at all.

The unexpected rewards.

Most notably our production build size of our primary swf went down by about 60%. The exact opposite was expected, simply because the debug swf size was so much larger. I attribute this to pure Adobe magic. With the reduction in size we were able to include a few more items into our primary swf in order to create a better user experience on the delay time of some of the other elements.

Of course one of the primary focuses of the Gumbo release was to make the designer happy (at the expense of the engineer sometimes). The ability to customize and make the product look just how you want has become much easier.

There have been may times when the words “Stupid Spark!” has been heard in our halls. In the end I think the changes are in the right direction and it is mostly to do with adjusting to a new system. We have began the discovery process to 4.5 now. Hopefully it is not nearly as painful.