recommendationRaccoon - My Personal Project
Two and a half weeks ago we had to choose personal project ideas. In a fortunate / unfortunate event, there were not enough client projects to go around. Apparently, client projects were typically undersubscribed until our class. In any case, I was forced to go out on my own to come up with cool project ideas. I honestly didn't have a problem coming up with decent ideas. One was a dating app that focused on 'tonight'. Ever had a night when you got off work but you didn't quite feel like going home? This app would allow you in a tinder like fashion to find someone within a mile of you with the intention of grabbing a drink or going on a mini date. Another idea was a marketplace for social currency. The premise being that there are many celebrities out there that are paid tons of money for their social media influence. One such example is Kim Kardashian whom is reportedly paid upwards of $10,000 per tweet because she's so influential on Twitter. The app I was going to build would be a marketplace to facilitate and broaden this market into mobile channels. I had a handful of other ideas like a site where guys bid on a date with a girl (think pageants) and all the money goes to charity. Another was a cohort-based weight loss app where users could band together and set a group goal with a social contract on Facebook. The list goes on.
Even after having all these half-baked ideas I realized the night before we were supposed to kick-off the project that none of these were any good. The point of the personal project was to sharpen our technical skills and building a simple consumer based product wasn't the best way to accomplish that. I thought day and night over the weekend to come up with something more technical and finally Sunday night I had an idea. I had really enjoyed the week our team skipped Ruby to learn about Machine Learning and Python but we had still not covered everything. One of the areas I really want to dive into was recommendations. I peeked around at other Github repos and there was only one recommendation engine on Node to speak of and it was very poorly documented. This was it! Despite my better judgment, I decided to go with this, it was the right mixture of exciting, unknown and useful. I also knew that I could easily build a nice demo to show off the engine and practice my front-end skills.
My journey with the recommendation engine began with a little bit of anxiety. I had only been exposed to machine learning for a week and even at that could barely code much of anything. One of the lead instructors at Hack Reactor, Shawn, told me that at the start I could easily get lost in research so that I should start out by building. I followed his advice and it really helped me get a grasp of the scope of the project. First step was to create an NPM module. 30 minutes later, done. Raccoon 0.0.1 was born. The name Raccoon had come to me as a running joke with my girlfriend and also Zack at Hack Reactor who helped us all fall in love with this random song about a woman being terrified of a raccoon outside her car. I digress. After finishing the creation of a barebones module, I set out to build the recommendation engine.
Shit, I didn't know how! After hitting this minor roadblock I dove into the minimal amount of research to get started. It consisted of watching the Recommendation Engine portion of the Stanford online course on Machine Learning (which I didn't find that helpful) then I dove into the Programming Collective Intelligence book that I had seen as recommended on Quora. This book was a goldmine. All the algorithms were written in Python but it was a great first start. The small anxiety I had related to the project began to slow. After slaving away for a couple days I had coding a basic collaborative filtering engine in Python and ported over the algorithms to Javascript. Great! I'm done! Haha just kidding. The engine in fact worked great but it was not built to scale. Scalability in a recommendation engine is one of the important factors. The thing is that my current implementation was query-based meaning that the recommendations were being created when the user asked for them. Not only that, it used a brute force analysis of every users movie ratings against every other user. If this doesn't sound scalable to you, you're right. I spent the second half of the first week starting work on the demo app for the engine. By the end of the week I had finished a styleless demo which enabled a user to enter in their ratings, enter a name, hit a submit button, and finally receive a ranked list of recommendations. All was well, except I still knew that the solution wasn't something that people could use at any scale (probably not over 100-1000 users).
At the beginning of the second week I had a big decision to make, finish optimizing the current implementation of the engine or scrap it and build something better. I began look around at other top recommendation engines and stumbled upon a few on Github that had found a lot of popularity. One of them was called Recommendable by David Celis. It was easily the best documented and user-friendly of the bunch. Interestingly he used Redis to store the ratings and recommendations. At the time I knew nothing about the benefits of Redis and even less about how to implement it. That said, I knew that other top engines used it and that Twitter used it for extremely demanding read/write tasks like their Timeline feature. I had hit another crossroads. This is where Hack Reactor really came in handy. Not in the advice I got in the moment about how to handle this. But in the advice Marcus had hammered into us from the start about how we should not be afraid to dive into project we know nothing about. And also the mentality I've started to adopt of constantly taking the harder route with regard to coding. Because frankly, the harder problems you tackle, the faster you learn. Also, things often turn out to be simpler than you originally thought.
Wednesday of last week I made the final decision to implement the engine on Redis. My reasoning was that Redis was a much better fit for recommendations because it could read/write at lightning speed (40,000 ops/second) which would enable my engine to build the recommendations on the fly whenever new ratings came in, instead of building them when users queried them. The trek began by reaching out. I already knew a little bit about recommendations now but I didn't know how to build a successful module or useful engine. I looked at all the top recommendation engines on Github and reached out the creator of the top one, Recommendable. To my surprise, David was extremely receptive to my questions. He wrote back a page and half after my first email which I was very humbled by. Here I was a student just starting with Javascript and a senior developer who made the most popular recommendation engine on Github was taking an interest in what I was building. We shot emails back and forth over the latter half of the week and having his support really helped me work through a lot of the struggles of how to structure the app. I kept at it until 3am basically every day last week and by Saturday at 11:30pm, I finally had a full working implementation of the engine (with bugs of course). This week i've been focusing on making it a solid module for other developers to use which involves creating test cases, thorough documentation and an impressive demo app.
Like most things at Hack Reactor, this was an extremely rewarding experience. I felt completely lost and confused on two major occasions but was able to keep moving forward. This project has certainly helped me understand how to be a good programmer and to be autonomous. Thank you to anyone who helped me along the way. Namely the Hack Reactor staff, students and David Celis.
And of course, here are the links to the repos and the demo:
Github of the engine: https://github.com/guymorita/recommendationRaccoon
Github of the demo app: https://github.com/guymorita/Mosaic-Films---Recommendation-Engine-Demo
URL of demo app: http://mosaic.nodejitsu.com
URL of NPM Module: https://npmjs.org/package/raccoon















