#skyknit

[Chunky Hat”, crocheted by Joannastar]

A while ago, I tried to train a neural network to generate knitting patterns by showing it a few thousand existing patterns so it could use trial and error to generate new patterns. I called the project SkyKnit, and its new patterns were… well, they were patterns. But they were nothing like the patterns it had been trained on. The test knitters of Ravelry’s adults-only, often-indecorous LSG forum had to do a lot of debugging to turn them into reality and, even debugged, SkyKnit’s patterns were strangely organic, prone to weird branching ribbing and organic-looking holes, turning into irregular shapes or even tentacles.

[Make Caows and Shapcho, knitted by MeganAnn, and Lacy 2047, knitted by michaela112358]

Ever since the SkyKnit project, I’ve wanted to do a version for crochet. Like knitting, crochet has strong ties to mathematics and programming, and in many ways it’s more flexible. A crochet algorithm could potentially create even more exotic shapes than SkyKnit did.

For this project, to be dubbed HAT3000, I decided to make some improvements. I would train it on just crochet hats, which ought to be simple enough and similar enough that a neural net might be able to figure out the general pattern and produce functional designs. The LSG forum crocheters helped me assemble training data of 500 vintage out-of-copyright patterns, plus these kind designers contributed their own patterns to the effort: krisitis-patterns, irishlacenet, SierraPelona, fairyhedgehogg, watrpriestess, RachyNewin, agnosticnun, UnplannedCauli, membril, Moogly, and SuviCrochets.

I used Max Woolf’s gpt-2-simple collab notebook to finetune OpenAI’s GPT-2-355M neural net model. It’s pretrained with a bunch of non-crochet knowledge from the internet (fanfiction, recipes, conspiracy theories), but it does have a much longer memory than SkyKnit does, so will have a better chance of being able to figure out how the rows relate to one another. HAT3000 was set up for success.

My first indication that something was going wrong was when the hats kept exploding into hyperbolic super-surfaces.

A pattern would start modestly enough with what could plausibly be the crown of a hat, but somewhere there would be an instruction that made the next row MUCH bigger than the previous row, and then the next row MUCH bigger than that. The later rows would turn floppy to fit all that length in while still attached to a much smaller row, and then the next rows would have to curl even more. Ruffles would turn to tight ruffles, and then to corals, and then to brains.

Here’s one rather innocently titled “Brim Hat Pattern #1708” which Ravelry user Persipan heroically crocheted all the way through the final row (or at least, halfway through that final row; we haven’t heard from them since, they’re probably okay). By Row 9, there are over 1700 stitches. Row 10 has 3500 stitches, took 9.75 hours to complete, and even that is nothing compared to the edging.

Almost all of HAT3000’s patterns did this, eating yarn, eating sanity, becoming more and more hyperbolic, threatening to collapse into black holes… why, since none of the example hats did this?

A few people explained it to me thus: It looks like it’s really hard for HAT3000 to NOT fill the planet with ruffly brains. Basically, since most hats are built from the center out, HAT3000 only knows how to make each circle bigger than the previous one, adding 1 new stitch for every 2 in the previous row, or, for the faster increases usually used at the center of a hat, as many as 5 or 8 stitches for every 1 in the previous row. That would still be okay if HAT3000 knew how to control its increases. Unfortunately, if it accidentally increases too much in 1 row, that excess increase gets multiplied by the next layer, which in turn gets multiplied again. The tiniest instability turns exponentially into yarn-eating disaster. Meanwhile, HAT3000 thinks it’s doing great because the pattern text itself is still very close match to what a human would do. It has no idea that its hat has exploded.

The worst offender is a pattern called simply “The End…” which by my calculations requires enough yarn to wrap the known universe in a ball of yarn about a billion light years thick.

Some of HAT3000’s patterns did actually produce hats, though they tended to all be tiny (probably because they had fewer rows and therefore fewer opportunities to begin suddenly expanding).

Other patterns produced not hats but strange artifacts, purpose unknown. Here’s the full pattern for one, called “Hang in Wind”:

Hang in Wind: Toss the pieces together. This is the windmill: Windmill in Wind: With a 1 cc ball of yarn, ch 14, join. Do not join, twist yarn to form a knot, or cut yarn.

Other patterns were equally inscrutable.

“New theory: HAT3000 didn’t want us to make hats for ourselves. We’re making hats for neural networks. Which are weird and amorphous and have dimensions and connections that humans cannot comprehend and also don’t really have heads.” - ScarlettTanager

One of the most entertaining aspects of the finetuned HAT3000 was how much of its original broad training it retained. Prompt it with the beginning of a story, and it could still add to the story in more or less the same style as the prompt. Hilariously, its strategy was now to gradually work the topic back toward crochet hats. In the paragraphs below, the prompt I gave HAT3000 is in bold. The AI did the rest.

Harry whirled to face Snape, heart beating hard. “Wh--What will you do?”

Snape’s eyes narrowed slightly and he lowered his wand, then stepped aside. As he approached behind Harry, he knelt down to kiss the front rim of Harry’s green-gloved hat as he pulled her closer. Snape slipped on the silk and reached in carefully, then unbuttoned Harry’s green-gloved hat. He looked Harry in the eyes and stroked her crown lightly before pushing on the seam of the hat to attach it. He finished on the front of Harry’s hat with a snap. With a light flourish he tied an extra strand of fabric, then another twist of silk, then another twist of silk.

HAT3000’s terrible crochet-themed erotica was even sometimes crochetable.

HAT3000 would sometimes turn its story prompts into things that were not crochet patterns exactly, but something strangely akin. Confrontations would turn into tightly choreographed fight scenes, the combatants weaving around and around each other. Villagers would build earthworks, walls, and moats in concentric circles. Recipes would turn into round cakes with dozens of stacked layers. Spaceships and adventuring heroes would follow complex and highly detailed routes. The trained model must have been able to draw some sort of connection between these various domains.

What have I learned from HAT3000? Like SkyKnit, it shows what unexpectedness happens when a neural net trainer, armed with approximately zero knowledge of a particular problem, nevertheless plows ahead and throws a neural net at it anyways. And like its knitting counterpart, HAT3000 owes its entire success to the creativity of human artists who took its often nonsensical patterns, fixed the bugs, and turned them into physical artifacts. I’d like to extend a huge thanks to the knitters and crocheters who sacrificed time, sanity, and yarn to satisfy the whims of these very weird AIs.

You can join the conversation, and crochet along, at the HAT3000 LSG thread (if you register first and don’t mind the swearing). There’s also a twitter thread that collects most of the patterns and many of the crocheted examples. There’s a big generated set of patterns here.

Subscribers get bonus content: I generated far more HAT3000 responses to story prompts than would fit in this post, but I picked some of my favorites to highlight as bonus material. I think they’re fascinating. If you have ever dreamt of a crossover between Star Wars/Harry Potter fan fiction and crochet, well YOU ARE IN LUCK.

You can order my book You Look Like a Thing and I Love You! It’s out November 5 2019.

Tried my hand at turning the SkyKnit pattern “Winder Socks” into actual socks, but didn’t think about my gague properly so they’re too big, but here’s what the pattern looks like! All those yarn overs give you a nice lacy pattern that’s hard to see on a varegated yarn, but it’s actually kinda pretty. 

[Make Caows and Shapcho - MeganAnn]

[Pitsilised Koekirjad Cushion Sampler Poncho - Maeve]

[Lacy 2047 - michaela112358]

I use algorithms called neural networks to write humor. What’s fun about neural networks is they learn by example - give them a bunch of some sort of data, and they’ll try to figure out rules that let them imitate it. They power corporate finances, recognize faces, translate text, and more. I, however, like to give them silly datasets. I’ve trained neural networks to generate new paint colors, new Halloween costumes, and new candy heart messages. When the problem is tough, the results are mixed (there was that one candy heart that just said HOLE).

One of the toughest problems I’ve ever tried? Knitting patterns.

I knew almost nothing about knitting when @[email protected] sent me the suggestion one day. She sent me to the Ravelry knitting site, and to its adults-only, often-indecorous LSG forum, who as you will see are amazing people. (When asked how I should describe them, one wrote “don’t forget the glitter and swearing!”)

And so, we embarked upon Operation Hilarious Knitting Disaster.

The knitters helped me crowdsource a dataset of 500 knitting patterns, ranging from hats to squids to unmentionables. JC Briar exported another 4728 patterns from the site stitch-maps.com. 

I gave the knitting patterns to a couple of neural networks that I collectively named “SkyKnit”. Then, not knowing if they had produced anything remotely knittable, I started posting the patterns. Here’s an early example.

MrsNoddyNoddy wrote, “it’s difficult to explain why 6395, 71, 70, 77 is so asthma-inducingly funny.” (It seems that a 6000-plus stitch count is, as GloriaHanlon put it, “optimism”). 

As training progressed, and as I tried some higher-performance models, SkyKnit improved. Here’s a later example.

Even at its best, SkyKnit had problems. It would sometimes repeat rows, or leave them out entirely. It could count rows fairly reliably up to about 22, but after that would start haphazardly guessing random largish numbers. SkyKnit also had trouble counting stitches, and would confidently declare at the end of certain lines that it contained 12 stitches when it was nothing of the sort.

But the knitters began knitting them. This possibly marks one of the few times in history when a computer generated code to be executed by humans.

[Mystery lace - datasock]

[Reverss Shawl - citikas]

[Frost - Odonata]

The knitters didn’t follow SkyKnit’s directions exactly, as it turns out. For most of its patterns, doing them exactly as written would result in the pattern immediately unraveling (due to many dropped stitches), or turning into long whiplike tentacles (due to lots of leftover stitches). Or, to make the row counts match up with one another, they would have had to keep repeating the pattern until they’d reached a multiple of each row count - sometimes this was possible after a few repeats, while other times they would have had to make the pattern tens of thousands of stitches long. And other times, missing rows made the directions just plain impossible. 

So, the knitters just started fixing SkyKnit’s patterns.

Knitters are very good at debugging patterns, as it turns out. Not only are there a lot of knitters who are coders, but debugging is such a regular part of knitting that the complicated math becomes second nature. Notation is not always consistent, some patterns need to be adjusted for size, and some simply have mistakes. The knitters were used to taking these problems in stride. When working with one of SkyKnit’s patterns, GloriaHanlon wrote, “I’m trying not to fudge too much, basically working on the principle that the pattern was written by an elderly relative who doesn’t speak much English.”

Each pattern required a different debugging approach, and sometimes knitters would each produce their own very different-looking versions. Here are three versions of “Paw Not Pointed 2 Stitch 2″.

[Top - ActualJellyfish; Middle - LadyAurian; Bottom (sock version) - ShoelessJane]

Once, knitter MeganAnn came across a stitch that didn’t even exist (something SkyKnit called ’pbk’). So she had to improvise. "I googled it and went with the first definition I got, which was ‘place bead and knit’." The resulting pattern is “Ribbed Rib Rib” below (note bead).

[Ribbed Rib Rib - MeganAnn]

Even debugged, the patterns were weird. Like, really, really nonhumanly weird.

“I love how organic it comes out,“ wrote Vastra. SylviaTX agreed, loving “the organic seeming randomness. Like bubbles on water or something,” 

SkyKnit’s patterns were also a pain. Michaela112358 called Row 15 of Mystery Lace (above) “a bit of a head melter”, commenting that it “lacked the rhythm that you tend to get with a normal pattern”. Maeve_ish wrote that Shetland Bird Pat "made my brain hurt so I went to bed." ShoelessJane asked, “Okay, now who here has read Snow Crash?”

[Winder Socks (2 versions) - TotesMyName]

“I was laughing a few days ago because I was trying to math a Skyknit pattern and my brain…froze. Like, no longer could number at all. I stared blankly at my scribbles and at the screen wondering what had happened til somehow I rebooted. Yup, Skyknit crashed my brain.” - Rayn63

[Paw chain 2 - HMSChicago]

On the pattern SkyKnit called “Cherry and Acorns Twisted To”:

“Couple notes on the knitting experience, which while funny wasn’t terribly pleasurable: Because there’s no rhythm or symmetry to the pattern, I felt I was white-knuckling it through each line, really having to concentrate. There are also some stitch combinations that aren’t very comfortable to execute physically, YO, SSK in particular.

That said, I’m nearly tempted to add a bit of random AI lace to a project, perhaps as cuffs on a sweater or a short-row lace panel in part of a scarf, like Sylvia McFadden does in many of her shawl designs. As another person in the thread said, it would add a touch of spider-on-LSD.” -SarahScully

[cherry and acorns twisted to - Sarah Scully]

BridgetJ’s comments on “Butnet Scarf”:

“Four repeats in to this oddball, daintily alien-looking 8-row lace pattern, and I have, improbably, begun to internalize it and get in to a rhythm like every other lace pattern.

I still have a lingering suspicion that I’m knitting a pattern that could someday communicate to an AI that I want to play a game of Global Thermonuclear War, but I suppose at least I’ll have a scarf at the end of it?” -BridgetJ

[butnet scarf - BridgetJ]

There was also this beauty of a pattern, that SkyKnit called “Tiny Baby Whale Soto”. GloriaHanlon managed somehow to knit it and described it as “a bona fide eldritch horror. Think Slenderman meets Cthulu and you wouldn’t be far wrong.”

[Tiny Baby Whale Soto - GloriaHanlon]

Other than being a bit afraid of Tiny Baby Whale Soto, the knitters seem happy to do the bidding of SkyKnit, brain melts and all.

“I cast on for a lovely MKAL with a designer I totally trust and became immediately suspicious because the pattern made sense. All rows increase in an orderly manner. There are no “huh?” moments. There are no maths at all…it has all been done for me. I thought I would be happy, yo. Instead, I am kind of missing the brain scrambling and I keep looking for pigs and tentacles. Go figure.” - Rayn63

Check out the rest of the SkyKnit-generated patterns, and the glorious rainbow of weird test-knits at SkyKnit: The Collection and InfiKnit. 

There’s also a great article in the Atlantic that talks a bit more about the debugging. 

If you feel so inspired (and don’t mind the kind-hearted yet vigorous swearing), join the conversation on the LSG Ravelry SkyKnit thread - many of SkyKnit’s creations have not yet been test-knit at all, and others transform with every new knitter’s interpretation. Compare notes, commiserate, and do SkyKnit’s inscrutable bidding!

Heck yeah there is bonus material this week. Have some neural net-generated knitting & crochet titles. Some of them are mixed with metal band names for added creepiness. Become a supporter of AI Weirdness to get more like these: