#moodsic

A few friends and I spent the weekend at a Microsoft-sponsored USC vs. UCLA hackathon. We worked diligently for 24 hours and made a pretty sick app that I’d like to share.

Also worth noting, our app actually ended up winning the overall hackathon. We barely scraped past the second-place team, who had a really killer idea as well. Either way, I’m super proud of my team and what we were able to accomplish, and I’m excited to share a bit of the experience with you.

The Team

For this hackathon, I teamed up with a couple of friends from USC, and our mission was to make ‘something dope’. We called ourselves SPDerp after our fraternity letters, Sigma Phi Delta. We sorta divided into front-end and backend, with Jesse Chand and Albert Kuo working on the front-end of the app while Daniel Silva and I crafted the backend.

Also a quick shoutout to Kabir Gupta who joined us for the Hackathon but due to team-size-restrictions couldn't be a part of SPDerp. He worked on his own dope application which you can check out here.

Being that the hackathon was sponsored by Microsoft, we were developing for either the Windows 8 or Windows Phone platforms. Since none of us had any experience in either of these platforms, we decided to make a web application instead. So our entire app is made up of native javascript and HTML, with a handful of calls to a couple of PHP scripts in cases where API calls could not be made through javascript due to cross-site-scripting security restrictions.

The Mission

So this hackathon had a theme : Mash-ups. Essentially we were asked to take two different APIs and mash them up together to create something interesting. It was a tough challenge since we’d have to learn how to use the API, how to get the info we need, and combine the APIs in an interesting way.

The Process

We started with Flickr. The truth is that I love Flickr. Where else can you get links to gorgeous, professional-quality images, with a brilliant indexing and querying system? But the real reason that I love Flickr is because it makes it easy to make an app look sexy. If you don’t believe me, take a look at the Yahoo! Weather app, which is almost certainly the most beautiful weather app that has ever existed.

Anywho, we decided that Flickr would be a good place to start, and so we started thinking about what we could do with a bunch of easily-queryable, high-quality images.

And, we thought, images can portray mood as well or better than words. Even an image of a landscape can resonate with a person on an emotional level. So perhaps we could analyze the mood of an image, get some basic mood data, and use that to do something interesting.

That something interesting, we decided, was to create a music playlist.

The App

We created Moodsic: a web app designed to create a perfect playlist by using analyzed image data to determine the person’s mood.

And since it’s all built with native javascript, I’ve hosted the app so that you can try it out for yourself. Quick disclaimer: We weren’t able to spend as much time as we’d have liked to on our image-mood-song analysis algorithm, so the generated playlists are somewhat arbitrary. But the logic is all there and it just needs to be fine-tuned to produce solid results (which we, of course, plan to do).

Feel free to give it a whirl– you will need to have Spotify open (either the native or web player) in order to actually play the generated playlist, but you get the gist.

The Process

If you’re not interested in any of the technical mumbo-jumbo/nitty-gritty/other-cliché, then you might want to skip this section.

Basically, we started by creating a native HTML/css/javascript environment. We designed it to look solid on a tablet (since it was to be demoed on a W8 tablet). In that light, it probably doesn’t look or feel too great on mobile.

The Images

So the Flickr API was easy enough. We decided to use a filter to only select images of landscapes, seeing as it’s easier to decipher the mood if there aren’t any actual people in the image. We were then able to restrict ourselves to very simple heuristics like average red, green, blue values, and brightness of the image.

We were able to extract these heuristics from the images by using the British Columbia Genome Sciences Center’s Image Summarizer API. We were originally set to use Imagga, but we weren’t quite able to get it to work and had to abandon it mid-hackathon.

Once we got the average red, green, and blue values for the image, as well as the images brightness, we were able to determine the ‘mood’ of the image. We got this by essentially mapping all of the possible combinations of different colors to some base emotion:

Low red, low green, low blue : dramatic

Low red, low green, high blue : calming

Low red, high green, low blue : epic

Low red, high green, high blue : reflective

High red, low green, low blue : angry

High red, low green, high blue : rebellious

High red, high green, low blue : aggressive

High red, high green, high blue : light

Admittedly, for this app to really work as desired, we’d have to incorporate some kind of machine learning algorithm as well. But for a 24-hour hackathon, this did just fine.

Also worth mentioning : based on the brightness of the color, we determined the minimum and maximum tempo (BPM) to search for. A bright image would return faster, higher BPM songs, while a darker image would return slower, low BPM songs.

The Songs

Once we had this image analysis on the three images that the user selected, we turned to the Echonest Song Search API to grab twenty random songs based on this information.

Finally, once we have a list of songs, we add each to a spotify playlist and display the music player.

An Interesting Challenge

I’d like to take a moment to explain one of the hackiest parts of our app: the php xml-jsonp wrapper that I built to allow us to query these XML based APIs from native-run javascript.

The challenge is that javascript queries are generally reserved for domain-level queries. Essentially, the javascript running on facebook.com can access any data or resource that is hosted on facebook.com. But if facebook needs to grab some Twitter data, for instance, it cannot do that through the client-side javascript, because this results in some crazy security issues.

So if I want to grab the 20 most dramatic songs from echonest via their api, I’d need some way other than AJAX to grab that information.

Enter jsonp, AKA json with padding. What I was able to do was use a very small PHP script (hosted on my site), to forward an incoming query to the Echonest API. Then, I’d wrap the resultant XML in a javascript method call, and then send it to the native app. This is allowed because the information must be in JSON-acceptable format, which restricts most cross-site-scripting vulnerabilities.

The Takeaway

This app was awesome to build. We wanted to make something really interesting that did some crazy magic behind the scenes to deliver awesome content, and an awesome experience to the user. We wanted to make it super enjoyable to use, and eventually we’d like to to learn how to make better predictions.

Either way, I think I speak for my team when I say that we are all humbled to have won the competition (and a free trip up to Seattle to visit Microsoft’s HQ). I personally had an awesome time, and I’m especially proud of how much we got done in the limited timespan of the event.

I told you to be patient/I told you to be fine/I told you to be balanced/I told you to be kind

Now all the love is wasted?/Then who the hell was I?/Now I'm breaking at the britches/And at the end of all your lines