#graphlab

I've had to sidebar my exploration of Bayesian statistics temporarily as things have come up. Having said that, the silver lining is that I've come across a few new (to me) tools that have transformed some of my usual analytical tasks.

dplyr (R)

This is the latest and greatest data manipulation suite from Yahweh of the R community, Hadley Wickham. Like many others, I had adopted the behavior of defecting from using the original plyr toolset when the number of categories became prohibitively large (usually on the order of thousands) by either utilizing another ghastly tool like aggregate() or pushing what I could into mapReduce, leaving much to be desired on the exploratory front. 

So in this regard, dplyr is a game-changer. Lightning fast grouping functionality, simply verb driven grammar for selecting, transforming, applying functions, sorting, subsetting...he's really thought of it all this time and the performance is simply marvelous. I'm never using anything else. 

The really great part is the built-in pipeline functionality. Specifically, by placing a "%.%" in between statements, you can create analytical sentences that flow from one operation to the next, very much like the "|" in UNIX scripting. For example, to get the top 20 home run hitters from the Lahman baseball statistics database:

> Lahman::Batting %.% group_by(playerID) %.% summarise(HR = sum(HR, na.rm = TRUE)) %.% arrange(desc(HR)) %.% head(20)

graphlab (python)

This a longtime incubating company out of seattle, and they've recently open sourced a series of python APIs to their scalable graph analysis engine. Yes, scaleable. Within a few minutes, a relative newbie to python even can start manipulating graphs using the relatively simply interface, generate user-item recommendations for applications such as the Netflix movie ratings competition or amazon style collaborative filtering algorithms (e.g., "people who viewed X also viewed Y") or finding the most important pages in a domain via the pageRank algorithm.

In fact, this appears to be the focus, as graphlab has an out-of-the-box set of recommender utilities, which are incidentally very simple - for example, the popularity algorithm is the simply exp(1)*P(item) - but they lay the foundation for integrating more sophisticated algorithms down the road. There are also allegedly some visualization tools that based on matplotlib (I think), but I have yet to make them work in an ipython notebook instance. Regardless, a lot of promise here and what appears to be an incredibly strong team.We'll be watching...

For a quick start, check out the 6 Degrees of Kevin Bacon tutorial (or should it be Danny Trejo?).

igraph (R or python)

As the people at graphlab seem to favor python for the time being, an R alternative that has been around a while and has many features of its own is igraph. I only really investigated the R library at this time, but apparently, there is a python analog. This tool is great for generating quick visuals of small graphs/subsets of larger graphs. The expression of edges or vertices takes some getting used to out of the gates, and it seems to be less recommender system oriented.

gridExtra (R)

This package extends grid, with some great convenience wrappers for building and publishing grobs to your graphics device. To date, my new favorites a grobTable() and grid.arrange(), the former to generate graphical tables without something like Sweave or knitr and the latter to arrange multiple grid panels with ease. No longer will I be looking up that code from the last time I had a multi-panel display not handled by facet_wrap(), instantiating a hacky viewport function.

>> (append in UNIX)

Robin Wauters for TNW:

Seattle startup GraphLab claims it is building the “fastest machine-learning analytics engine for graph datasets”, based on the popular open-source distributed graph computation framework with the same name, and it has just raised capital to come through on its promise.

Good luck to GraphLab’s team.

✚ Here’s a short list of MapReduce implementations for graphs.

Given Enterprise's are still not sharing data at scale, the case for relational databases still holds water, yet Graph Database technologies could bring  game-changing productivity seen in online social networking.

Consider Bill of Material applications contain many lists with nested graphs. SAP HANA acknowledged gleaning business intelligence slow from nested graphs and ill fitted for today, adopting in memory databases. But consider  what the GraphDB could usher in. Super opportunity to learn at GraphLab WorkShop ML Conf  on large scale machine learning.