Appmine.io

For most of our processing, we are using Pig on top of Hadoop. To make things easier, CDH4 provides enough packages for us to get started fast.

Our views are stored in ElasticSearch, which means we need a way to read information from HDFS, process it in Pig and store the result in ElasticSearch. Luckily enough, the kind people at InfoChimps have a project on their github called Wonderdog.

Wonderdog contains an ElasticSearchStorage which we can use for reading and storing information from pig into ElasticSearch as we are used to.

Now for the catch.

We are using CDH4, which isn't supported by Wonderdog. Another downside is the fact that there is a nasty side-effect when you want to write multiple relations to ElasticSearch in a single pig script.

Since we really need this tool, we decided to patch wonderdog and commit the sources back to the community (https://github.com/calmera/wonderdog) Basically we did the following:

- Added CDH4/hadoop2 as a dependency to wonderdog - Upgraded the ElasticSearch dependency to 0.19.4 - Made the schema definitions which are stored in the UDFContext aware of the context signature.