data kitchen

<p class="p1">Rubbish in rubbish out is the popular idiom with data analysis. The vocabulary needs to be of high quality for the classifier to sniff out sin well. The first problem is the use of a one word vocabulary. We speak in sentences and phrases - cake may be associated with gluttony, but the phrase 'more cake' has a stronger resonance. I decided to use a natural language package from Apache, opennlp to decompose the text into its parts of speech (adjective, noun etc). The terms most indicative of the text I decided to be the noun and verb phrases. I also chose to isolate the adjectives, as sins themselves are of descriptors as well as subjects and objects.</p><p class="p2"><br /></p><p class="p1">The claasifier in weka uses a StringToWord vector to store the instances of words in a document. The StringToWord is a filter that can be applied to input data to transform it into a format appropriate for classification. The format involves creating a matrix of features, features are aspects of the data that can be used to predict the class. In our model here, have been using the appearance of words, so the filter would break down all the words in the training data, and compute the count of each one. The filter uses a tokeniser to break the document into words, I replaced the tokeniser with a phrase tokeniser that uses opennlp to extract the most salient parts of speech, thus we are now tracking the counts of phrases as opposed to words, where we would be catching a lot of noise.</p><p class="p2"><br /></p><p class="p1"><br /></p><p class="p1"><br /></p><p class="p1">Code here.</p><p class="p2"><br /></p><p class="p1">I also changed the money seed vocab for the vocab generation to words describing money in a positive light, 'wealthy', 'affluent' to try and identify the love of money better. The phrase extractor is injected into the mapping phase to create a like for like classification.</p><p class="p2"><br /></p><p class="p1">The stream graph used earlier is nice looking, but has a habit of conflating results. The dominance of the reduction of lustiness overshadows the observed behaviour of envy which seemed to remain fairly constant. So I decided to show the sins in isolation, the JavaScript is hacky, but I wanted to focus more on the sin extraction code! All sat in githhb if you fancy a migraine.. </p>

Rubbish in rubbish out is the popular idiom with data analysis. The vocabulary needs to be of high quality for the classifier to sniff out sin well. The first problem is the use of a one word vocabulary. We speak in sentences and phrases - cake may be associated with gluttony, but the phrase 'more cake' has a stronger resonance. I decided to use a natural language package from Apache, opennlp to decompose the text into its parts of speech (adjective, noun etc). The terms most indicative of the text I decided to be the noun and verb phrases. I also chose to isolate the adjectives, as sins themselves are of descriptors as well as subjects and objects.

The classifier in Weka uses a StringToWord vector to store the instances of words in a document. The StringToWord is a filter that can be applied to input data to transform it into a format appropriate for classification. The format involves creating a matrix of features, features are aspects of the data that can be used to predict the class. In our model here, have been using the appearance of words, so the filter would break down all the words in the training data, and compute the count of each one. The filter uses a tokeniser to break the document into words, I replaced the tokeniser with a phrase tokeniser that uses opennlp to extract the most salient parts of speech, thus we are now tracking the counts of phrases as opposed to words, where we would be catching a lot of noise.

Code here.

I also changed the money seed vocab for the vocab generation to words describing money in a positive light, 'wealthy', 'affluent' to try and identify the love of money better. The phrase extractor is injected into the mapping phase to create a like for like classification.

The stream graph used earlier is nice looking, but has a habit of conflating results. The dominance of the reduction of lustiness overshadows the observed behaviour of envy which seemed to remain fairly constant. So I decided to show the sins in isolation, the JavaScript is hacky, but I wanted to focus more on the sin extraction code! All sat in githhb if you fancy a migraine.. 

Latest results

If the data tells the truth, it has smacked me in my money cussing face. Here is the latest breakdown. On the good side, there is a strong trend observable here, the more money is mentioned, the less sinful the text becomes. Possible theories upon why include the nature of money based text is more formal and less likely to include actively expressive language. Plus its only run on a sample of the crawl. I may pretend its particularly money-pious sample.

Frantically trying to justify the need for my own satellite..

We're only pretending to be a civilised species. The internet is basically a massive porn cake with a microscopic membrane of lucid content on top. To improve my sin quotients I decided to apply a machine learning classification algorithm. The broad idea is that you train the classifier using preclaasified documents to help it learn how to classify unclassified documents.

As with all machine learning, the more data, the better. Lots of data still outperforms sophisticated algorithms. Excellent write up by Peter Norvig here. So my next task was to generate a corpus of data for each sin and money to train the classifier. This raised an interesting modelling problem of how to define sin within a text. Essentially I wanted the text itself to be sinful, this meant that others would describe the text with the sin. Therefore I decided to make use of tagging, pulling down urls from bookmarking sites such as delicious and diigo that were tagged as each sin. This is where I discovered the porn invasion. The trouble with crawling the links was turning up sparse useless documents (example).

Data cleansing a boring and dull and vital task. The class I created checked for repetitious content (spam) and for sequences of one sentance paragraphs, as from observation this indicated menu items of videos of rather 'erhem' content. This provided a very decent cleanse of the data, and left me with roughly 15% of my original content. Relevant method below.

5 billion crawled web pages is a treasure trove of data, so when commons crawl launched a contest with their release of the 2012 corpus, I decided to give it a bash. A massive web crawl will yield very broad data, so I decided to seek general patterns. I asked myself what I would want to know about humanity. Simple enough? Looking at what's been happening this past year, austerity came to mind, which got me thinking about money and how it affects us. From here I decided to try an answer the age-old question - is money the root of all evil?

Having only left myself an evening to enter, I took a simple brute-force approach which I hope to evolve with more sophisticated nlp techniques and random heuristics (text is super messy). The code is a hadoop mapreduce task, the mapper maps the 'moneyness' of a webpage. The moneyness is a measure of how much the document is concerned about money. To calculate this I mocked up a simple vocabulary, and calculated the inverse document frequency of term matches. (matchedTermCount / totalWordCount). I then did the same for the seven deadly sins, calculating a sin vector for each document made up of lust quotient, greed quotient etc.

The output keys of the reduce task was the moneyness score. This allowed the reducer to calculate averages of the sin vectors for increments in moneyness. Hopefully showing an increase in sin alongside money.

The map reduce job would produce output vectors which even in my nerdiest mood, I'd not want to read, so I decided to construct a visualisation. I use Mike Bostock's excellent d3 visualisation library. It's a very flexible javascript library that a lot of talented folk clearly use as the examples page is a bomb of amazing visualisations. I settled upon a streamgraph as a good way to view the changing amounts of sin across moneyness. Results coming up!