#twitter corpus

A really interesting post by Rachael Tatman about emoji and information theory. Excerpt: 

Information theory is the study of storing, transmitting and, most importantly for this project, quantifying information. In other words, using an information theoretic approach we can actually look at two input texts and figure out which one has more information in it. And that’s just what we’re going to do: we’re going to use a measure called “entropy” to directly compare the amount of information in text and emoji. [...]

So if you have a string of text that’s just one character repeated over and over (like this: 💀💀💀💀💀) you don’t need a lot of extra information to know what the next character will be: it will always be the same thing. So the string “💀💀💀💀💀” has a very low entropy. In this case it’s actually 0, which means that if you’re going through the string and predicting what comes next, you’re always going to be able to guess what comes next becuase it’s always the same thing. On the other hand, if you have a string that’s made up of four different characters, all of which are equally probable (like this:♢♡♧♤♡♧♤♢), then you’ll have an entropy of 2.

TL;DR: The higher the entropy of a string the more information is in it. [...]

Here’s the density (it’s like a smooth histogram, where the area under the curve is always equal to 1 for each group) of the entropy of an equivalent number of emoji spans and tweets.

Text has a much high character-level entropy than emoji. For text, the mean and median entropy are both around 5. For emoji, there is a multimodal distribution, with the median entropy being 0 and also clusters around 1 and 1.5.

It looks like my hypothesis was right! At least in tweets, text has much more information than emoji. In fact, the most common entropy for an emoji span is 0: which means that most emoji spans with a length greater than one are just repetitions of the same emoji over and over again.

Read the whole thing. 

A Book of Explanations (NaNoGenMo 2016)

NaNoGenMo 2016 has concluded, and there are dozens and dozens of new computer-generated novels to share with you.

Thanks to Ranjit “Moonmilk” Bhatnagar, we finally have an explanation for that question that’s been troubling you.

You can an explanation of how he did it on the project’s git repository, but the thing I really like here is how often the metre of the phrases meshes with the accidentally perfect answer. It works as a poem. The staccato rhythm of repeating “why...because...” is modulated by the rhyming.

Being sourced off of Twitter, some of the questions-and-answers are about as non-worksafe as you might expect, and the answers veer from the profound to the disturbed...