My Poison Pen

There’s a game called Inktober where people post one drawing for every day in October. To help inspire people, the people behind Inktober post an official list of daily prompts, a word or phrase like Thunder, Fierce, Tired, or Friend. There’s no requirement to use the official lists, though, so people make their own. The other day, blog reader Kail Antonio posed the following question to me:

What would a neural network’s Inktober prompts be like?

Training a neural net on Inktober prompts is tricky, since there’s only been 4 years’ worth of prompts so far. A text-generating neural net’s job is to predict what letter comes next in its training data, and if it can memorize its entire training dataset, that’s technically a perfect solution to the problem. Sure enough, when I trained the neural net GPT-2 345-M on the existing examples, it memorized them in just a few seconds. In fact, it was rather like melting an M&M with a flamethrower.

My strategy for getting around this was to increase the sampling temperature, which means that I forced the neural net to go not with its best prediction (which would just be something plagiarized from the existing list), but something it thought was a bit less likely.

Temperature 1.0

At a temperature setting of 1.0 (already relatively high), the algorithm tends to cycle through the same few copied words from the dataset, or else it fills the screen with dots, or with the repeated words like “dig”. Occasionally it generates what looked like tables of D&D stats, or a political article with lots of extra line breaks. Once it generated a sequence of other prompts, as if it had somehow made the connection to the overall concept of prompts.

The theme is: horror. Please submit a Horror graphic This can either be either a hit or a miss monster. Please spread horror where it counts. Let the horror begin… Please write a well described monster. Please submit a monster with unique or special qualities. Please submit a tall or thin punctuated or soft monster. Please stay the same height or look like a tall or thin Flying monster. Please submit a lynx she runs

This is strange behavior, but training a huge neural net on a tiny dataset does weird stuff to its performance apparently.

Where did these new words come from? GPT-2 is pretrained on a huge amount of text from the internet, so it’s drawing on words and letter combinations that are still somewhere in its neural connections, and which seem to match the Inktober prompts.

In this manner I eventually collected a list of newly-generated prompts, but It took a LONG time to sample these because I kept having to check which were copies and which were the neural net’s additions.

Temperature 1.2

So, I tried an even higher sampling temperature, to try to nudge the neural net farther away from copying its training data. One unintended effect of this was that the phrases it generated started becoming longer, as the high temperature setting made it veer away from the frequent line breaks it had seen in the training data.

Temperature 1.4

At an even higher sampling temperature the neural net would tend to skip the line breaks altogether, churning out run-on chains of words rather than a list of names:

easily lowered very faint smeared pots anatomically modern proposed braided robe dust fleeting caveless few flee furious blasts competing angrily throws unauthorized age forming Light dwelling adventurous stubborn monster

It helped when I prompted it with the beginning of a list:

Computer Weirdness Thing

but still, I had to search through long stretches of AI garble for lines that weren’t ridiculously long.

So, now I know what you get when you give a ridiculously powerful neural net a ridiculously small training dataset. This is why I often rely on prompting a general purpose neural net rather than attempting to retrain one when I’ve got a dataset size of less than a few thousand items - it’s tough to thread that line between memorization and glitchy irrelevance.

One of these days I’m hoping for a neural net that can participate in Inktober itself. AttnGAN doesn’t quiiite have the vocabulary range.

Bonus content: An extra list of 31 prompts sampled at temperature 1.4. I’m also including the full lists of prompts in list/text-only format so you can copy/print them more easily (and for those using screen readers). Enter your email here to get them!

And if you end up using these prompts for Inktober, please please let me know! I hereby give you permission to mix and match from the lists.

Update: My US and UK publishers are letting me give away some copies of my book to people who draw the AInktober prompts - tag your drawings with AInktober and every week I’ll choose a few people based on *handwaves* criteria to get an advance copy of my book. (US, UK, and Canada only, sorry)

In the meantime, you can order my book You Look Like a Thing and I Love You! It’s out November 5 2019.

Now that algorithms like GPT-2, BERT, and GROVER have trained on a big chunk of the internet and are powerful enough to hold a train of thought for more than a sentence or two, people are worried that it might become easy to generate large quantities of hard-to-moderate fake news.

I generated the above using GROVER’s fake news generation/detection demo. As you can see, it’s still not too difficult to spot neural net-written text. But they’re releasing demos like this because it’s important to learn how good it can be, and how to build algorithms to detect it.  For a careless reader who only reads the headline (not that that’s any of us) or a site trying to rise in search rankings through sheer volume of unique articles, neural net-generated nonsense may be a substitute for real information. It’s also easier than ever to produce fake screenshots of articles in support of your argument.

The GROVER demonstration is at its most convincing when generating political news or blog posts.

But my favorite thing is to give it a highly improbable headline and watch it just try to roll with it.

Eli the dragon is a JERK.

You can generate your own fake news article at https://grover.allenai.org/

I generated more articles than would fit in this blog post. To get them (and sign up for future bonus content), enter your email here!

In case you missed it: you can preorder my book!

Amazon - Barnes & Noble - Indiebound - Tattered Cover - Powell’s

People send me the weirdest ideas.

I knew that fruit flies are a mainstay of research labs, but I had never given them much thought until Prof. Greg Neely emailed me to point out how weird the names of mutant fruit flies are. There’s a strain of mutants called “Out Cold” where the fly loses motor coordination below a certain temperature, and another nicknamed “Moonwalker” that walks backwards. Could a neural network learn how to invent new names and mutations?

The problem was I didn’t have a big list of examples to train a neural net on - there just aren’t that many quirky ones. So I turned to a strategy I’ve used in the past: prompting a general-purpose neural net called GPT-2. GPT-2′s training examples were huge chunks of text from the internet - maybe it had seen enough on the internet that it could figure out what I was asking it for? I prompted GPT-2 with a short list of example fruit fly mutants, and asked it to add to the list. The smallest GPT-2, the 117 million parameter model I sampled via Max Woolf’s GPT-2-small interface, wasn’t quite up to the task - it would drift into song lyrics or conversations or lists of vampires. The next-largest version of GPT-2, however, the 345 million parameter model I sampled via talktotransformer.com, often got the hang of it. I’m particularly impressed at how the names seem to match their attributes, even if the attributes are a bit far-fetched.

Macho Man: Likes to punch.

Soap-Sucking Insects: a gene that causes the fruit flies to be attracted to soaps which smell funny.

Tornado: A fly that can take a huge amount of damage during turbulence.

Chocolate Eye: The fly that drinks and stumbles when eating candy.

Jungle Fire: Any of several genes that makes the flies more prone to fires.

They get weirder. A lot weirder.

Jingle Bells: Does not speak but sings.

The Last Emperor: Has a huge black mustache.

Unicorn Faced: A gene that prevents flies from developing fruit fly traits.

Grigk: A gene that makes the fruit fly think it has been turned into a gorilla.

Frogman: The fruit fly that makes a nuisance of itself when you try to do anything dirty.

Lover-Possum-Worm: A gene that affects mating habits of fruit flies.

The Brain-less Fly: The fly that has no brain.

I’m with the Cheddar: The fly that has no brain. But the flies aren’t with the Cheddar!

Pork and Cheddar: The flies where the hair was replaced by the brains of those who ate Pork and Cheddar.

Some of the fly mutants are rather terrifying. I think you might need to fill out a special form to be allowed to use these mutants.

Linda: Is able to communicate through vibrations.

Zebra: Gives off sparks when he moves.

Lana: Her eyes are covered by a large, purple cloud.

Nerd Rage: Has two eyeballs, but is pretty much invincible.

Chocolate Corn Cone: Has six heads. It’s made out of chocolate and corn (in addition to the brains from its victims).

Lemondogs: The fruit flies from the American Southwest who were responsible for killing the dinosaurs and driving scientists crazy years earlier. As a consequence they evolved into the same kind of insect that scientists now recognize today as the Lommy Beetle, the “mammal-sized flea” from the Southwest.

The Worm: Gives birth to worms and can fly. Wormholes between worlds and hyperspace allow the worm to travel to and from one world and return to the home (and body) of another.

These are some of the subtler mutants. Takes careful experimentation to figure out which one you have. At least you can be pretty sure it’s not Chocolate Corn Cone.

Maggie’s Kiss: A fly that doesn’t even

In a Perfect World: A fruit fly that’s never met and never will be.

I’ma Be Here Now: Causes people to be happy even when they’re still alive.

The Cat that Kicked the Can: Is a good boy.

Cute Silly: A fly that is cute and lovable when it is stressed.

Gripping Burden: Stays alive even if it doesn’t move much.

Last year I trained a neural net to generate new names for kittens, by giving it a list of over 8,000 existing cat names to imitate. Starting from scratch, with zero knowledge of English or any context for the words and letter combinations it was trying out, it tried to predict what letters might be found in cat names, and in which order. Its names ranged from the strange to the completely nonsensical to the highly unfortunate (Retchion, Hurler, and Trickles were some of its suggestions). Without knowledge of English beyond its list of cat names, it didn’t know what letter combinations to avoid.

So I decided to revisit the cat-naming problem, this time using a neural net that had a lot more context. GPT-2, trained by OpenAI on a huge chunk of the internet, knows which words and letter combinations tend to be used together on the English-language internet. It also has (mostly) figured out which words and letter combinations to avoid, at least in some contexts (though it does tend to suddenly switch contexts, and then, yikes).

When I trained GPT-2 on the list of cat names using Max Woolf’s colab notebook, it still retained a lot of what it had learned from the rest of the internet. Gone were the strange names like “Tilly-Mapper” and “Balllucidoux” - it had a bunch of real words it could use instead. Here are some of the names it came up with - and the Morris Animal Refuge (who you may remember from that time they used neural net names for their guinea pigs) has given some of these names to some highly adoptable kittens.

First, neural net can do fancy:

Taffeta Pompompur Monocle Tom Glitter Notion Tinnitus Cheesemonger M. Tinklesby Linklater Soap

It can also do the opposite of fancy:

Scat Cat Butthole Gangrene Moisture Grotesque Petard Oilbag Buttwig The Cream Meatbag Dr Fart Fudge Putty Scumbag Constipation BUTT

And it can also do names ranging from tough to downright sinister:

Miss Vulgar Lillith The Vamp Elle Fury Deadbolt Romeo of Darkness Starmaker Fist Warning Signs Bibles Smoked The Firestarter Higher Rune Scarlet Be Thy Coat Kill All Humans Bones Of The Master Mr. Sinister Evil Whispers Spawn Serendipitous Kill Stranglehold

(Starmaker and Sparky Buttons are from a litter that had upper respiratory infections that damaged their eyes, but even though their world is kinda cloudy, they love to play and cuddle.)

I’m a particular fan of the Very Weird cat names:

Honeystring Dr Leg Tom Noodle Pinball Scene Peanutbutterjiggles You’re Telling A Lie Beep Boop Thoughts Bobble Bun Atmosphere You Name It Whiskeridoo Sparky Buttons

Seemingly This Guy Various Authors Chicken Whiskey Fish Especially Thelonious Monsieur Ringo Shuffles Sweet Cakes EXTAs (Eye Stalks) Checker Spin Donut Quin Two Patz Grandpa He Glad Funky Moe Fluttering Feelers Accepted A Tribute Chewie Bean PLEASE Gregory Chimney Notable PRODUCT LEGEND Weird Science Platinum Not Suitable For Character the Enforcer

Did I mention these cats are adoptable? If you live near Philadelphia, you live near these kitties!

32. We’re putting you in charge of Supernatural canon for the day. Paint a cannon with an elaborate SPN mural showcasing something that you think should be canon, with a caption like, “It’s #SPN Cannon!”

(It also looks pretty damn good in the dark!)

cops

soldiers

corporations 

homophobes & transphobes 

racists

assholes who fetishize lgbt people and think we’re all weird party animals and treat pride as if it’s a zoo 

1) all cops are bastards

Justin Trudeau is dead to me.

Do you hear that? That’s the sound of Liberal votes evaporating in B.C. and Environmentslists, and Indigenous people who have been betrayed again.

Justin Trudeau approved an oil pipeline one day after their government passed a Climate Emergency Motion.

What a spineless neoliberal fraud.

No matter who approves it, this pipeline will not be built,” said Will George, a leader of Protect The Inlet and community member of the Tsleil-Waututh Nation whose territory encompasses the tanker terminal. George and thousands of supporters with Protect The Inlet led public resistance to Kinder Morgan, causing mass arrests for six months last year outside Trans Mountain facilities. The Indigenous-led opposition has been ramping up recruitment for more resistance in anticipation of today’s approval with the launch of https://www.stoptransmountainpipeline.org/.

“People in British Columbia are the ones risking disaster from spills and we’re prepared to do whatever it takes to stop this pipeline,” George continued. “Tonight in downtown Vancouver we’ll show our opposition to this destructive pipeline and on Saturday we will further escalate our actions.”

This Saturday, June 22, Protect the Inlet and George will be in Victoria for a 22 kilometre walk where local activists have been building a Tiny House to support the Secwepemc Tiny House Warriors, who are preparing for land defense in the Interior of British Columbia. 7km of Saturday’s march will occupy the highway, before delivering a completed Tiny House to Island View Beach.

British Columbians oppose the project because it lacks consent from many Indigenous peoples and Nations along the route and tanker radius; a tanker spill jeopardizes endangered whales, salmon and other ocean life while threatening local jobs in tourism, fishing and other coastal activities; and the expansion of fossil fuel infrastructure and subsequent climate impacts are inconsistent with the climate emergency recently declared by the federal government.

Last year, more than 200 people were arrested at Trans Mountain construction sites before the Federal Court of Appeal ruled in favour of First Nations and environmental organizations, overturning permits and halting construction. Pipeline opponents expect arrest numbers to rise significantly when construction begins in earnest.

Canada declared a climate emergency on June 17, which is largely seen as a meaningless gesture given today’s decision on Trans Mountain. The pipeline’s new upstream (production and transportation) climate impacts would add an additional 21- 27 million tonnes of upstream greenhouse gas emissions annually to Canada’s total, roughly equivalent to 4,458,599 new passenger vehicles on the roads every year, or cutting down 24,715,361 acres of forest in a year, as calculated by the EPA Greenhouse Gas Equivalencies Calculator: https://www.epa.gov/energy/greenhouse-gas-equivalencies-calculator

Despite the federal government’s ownership of the pipeline and supposed commitment to climate-safe policies, there has been no formal accounting of climate pollution from the Trans Mountain pipeline and tanker project. A study commissioned by the City of Vancouver determined the downstream (end user burning) climate impacts of additional greenhouse gas pollution to be 71.1 Mt Co2e (carbon dioxide equivalent) annually, roughly equivalent to adding 13,694,444 passenger vehicles for one year or cutting down 75,912,447 acres of forest in a year.

like mr. billionaire with the 8th yacht this week is going to preach to poor people buying a new couch they probably didn’t need but wanted because their kids fucked up the other one.

rich person who underpays said poor person: you know you can save hundreds of dollars just by not buying all your coffee at cafes.

poor person: i genuinely don’t know what planet you live on but i do not buy coffee at a cafe.

rich person who knows full well all jobs require you have a phone: just sell your phone. you know … something that will get you money once and then you’ll never be able to replace it… effectively losing yourself money.

rich people don’t need a 7th yacht to make themselves happy but like … a poor person struggling under crushing poverty buying something that makes them happy is always a more justifiable expense.

After 20 painstaking years of research, Eva Ramon Gallegos, a Mexican scientist has finally developed a cure to eliminate 100 percent human papillomavirus and prevent the spread of cervical cancer among women. (x) (x) (x) Y’all they cured HPV

Why do all the black hash tags no longer exist?

And yet we still have all of these…