#underfitting

[You thought I forgot about you, didn’t you?  It’s a rough time for everyone right now, and so, particularly recently, it’s been easy to let projects like this fall by the wayside.  It’s hard to argue that this blog is as important as, say, protesting the illegal detainment of lawful American residents.  But, as a friend of mine said: we’re fighting to be able to live the lives we want, and living well is a form of resistance, too.  Besides: hip hop often gives a voice to the oppressed. It’s a predominantly Black art form with undeniable Muslim influences that was pioneered by an immigrant.  If there was ever a time to celebrate this kind of art, well...it’s now.

I’ll try to post once or twice a month (so: every 3ish weeks) moving forward.  In the meantime, here’s a post that I somehow forgot about until now...]

My last post talked about the neural network that I built to generate novel hip hop lyrics.  If you want a rough overview of how that model works, read it.  This post talks about how the model performed during different stages of the training process, and how I worked to improve it.  

Recall that neural networks train through repeated exposure to data.  They start out by giving each feature, and each interaction, a random weight.  Each time the model makes a prediction, it evaluates whether or not that prediction was “right” (i.e., that the predicted output matches the actual output) and adjusts the random weights so that future predictions will be more accurate, using (something like) gradient descent.  This is why, broadly speaking, models that have been trained on larger data sets, and for longer periods of time, perform more accurately.

I actually wrote and trained several models before settlings on the final version, and I want to start out looking at the first draft.  This model was trained on a (very small) dataset of 100k characters, because that was the limit of what my computer could nimbly handle.  Does 100k characters should like a lot?  It isn’t.  I started with a relatively simple model -- metaphorically, it was equivalent to a brain with a smaller number of neurons.  Generally: fewer neurons mean a simpler hypothesis, but also faster training times.  If you want to know more about how the number of neurons (or, units) in a model can contribute to the model’s predictive power, The NYTime’s write up of Google Translate has a really good overview.

Because of the kind of model I chose (a LSTM-RNN) and the library I used to implemented it (Keras) my model trained in epochs -- full passes through the dataset.  After each epoch (in this case exposure to all ~100,000 patterns of 101 characters), I established a “checkpoint” that stored the weights associated with the model at that point.  Then I ran the model with those weights.  I used the outputs of these runs (as well as, you know, actual mathematical accuracy metrics) to determine how well the model was doing after each stage of training.

Here’s how the first model did after 5 training epochs.  Again, [seed text] is bolded inside brackets.  Model generated text follows.  Are you ready?

[similar to saying mama s baby s daddy maybe  when we had sex i was in the mercedes a]nd when i cac  bucy mh in i m ne bor   oht   duai  se to me   mh bol mh  oo  i  oh the  loal oh  i  tou pan a bia l a n a lii  io she bar h gon t gan  th toe datche ...

Ouch. It’s halfway between English and...Vietnamese? Obviously, we’re not doing very well right now.

Here’s how the same model looked after 15 training epochs:

[i m from the belly of the beast remember i barely used to] lame the toak   mo  bot t ale the toie     aaasnz  i make her dance thas sae io toe tooe the toal [...] what happened shas iapp nn thet the saad  i mote the bitch  i mote the siie [...] i want a whip and a chain i want a whip and a chain

Medium ouch.  It almost starts out strong: the first word it produces is actual English, even if the sentence remember I barely used to lame isn’t so promising. Then we get total gibberish.  About halfway through, we get a little more actual English (what happened) mixed in with gibberish.  At the end, it looks like something promising is happening, but...

...Here’s the model after the full 30 epochs:

[that cr nack  yeah i got cr nack started from the trap  now i rap] this the shit you play when you sipkin suck it up this the shit you play when you snoke a zip and up this the shit you play when you sippin  out a cup [...] this the shit you play when you sippin  out a cup this the shit you play when you sippin  out a cup this the shit you play when you sippin  out a cup [...]

What looked promising after 15 epochs was really a disaster after 30: the model was basically memorizing phrases that were disproportionately common (because, say, they were part of hook of Bentley Truck) and rewarding itself for always predicting those phrases.  

Part of the problem was the dataset -- I trained this practice model on too small a sample.  But the way the model was built contributed to this problem, too: Always outputting the same result is an underfitting (bias) problem.  Basically, this was a symptom that the model was too simple.  As a result, the hypothesis that the model held about how to generate language was also too simple to be insightful.  Normally, a model that’s “too simple” is a model that doesn’t consider a large enough number of features, or that isn’t looking at enough feature interactions (because, say, it doesn’t have enough neurons to do so).  Because my number of features was set (the number of unique characters in my dataset) I dealt with this problem by increasing the number of interactions the model looked at.  Basically, I made the model better by giving it more neurons.

Here’s how the final draft (i.e., the more complex version) of the model performed during training.  This obviously isn’t a perfect comparison, because the final model was also trained on a larger dataset, but still:

After 5 epochs:

[diamond rings you say i m bad at timing things so what s a man to do when all i hand to you is handed] soees in men mo more to matee ien so take the brain out  leave the heart in take the brain out  leave the heart in bany mote it paf the bott to the bott to the back in the same  io a saaee  the same the same  i m the one the peelen the cooper so the [...]

It looks like English! Some of the repetition/memorization in the middle (take the brain out  leave the heart in take the brain out  leave the heart in) looks like the same underfitting problem we saw earlier, but it’s less overwhelming.  And this is still the beginning of the training cycle.

After epoch 15:

[motherfucker tis the season  to be servin what you doin    mob mobbin like a motherfucker] e saaee the siie the saaee the siig she s soal the boas  sook doon  i meed mo mnee in the middle  bou   a b t in the way i maae  i m gonna find a way to make it without you i m gonna hond in you ra babk to the boat   rock the boat  change positions new position  new position  new position stroke it for me stroke it for me stroke it for me [...]

So far, still promising.

And finally, after the last training epoch:

[you should have listened motherfucker when i said don t get it twisted  don t get it twisted  nigga] het said  i m the one to gat  mo money more to work to mess with me me   yeah  yeah  yeah  yeah  yeah  yeah  yeah  [...] let me see you can t say the same   i m gonna be alright   see the same black out  black out   black out black out  black out black out  black out