#pix2pix

There’s a kind of neural net that will convert block drawings into its best attempt at a photorealistic scene. Now it’s easier than ever to try them out, without any coding or fancy computing equipment needed. Today I’m going to show you an algorithm developed by Nvidia called SPADE. There’s an online SPADE demo called GauGAN, and runwayml includes TWO versions of SPADE with their latest free beta. Here’s an example of what you can do with them: I used SPADE-COCO from runwayml to turn this landscape:

into this fully-shaded scene. Notice how generous the neural net was in converting my simple boat and cloud shapes into something more realistic. It even decided where to add reflections.

With some of these demos, you can do only landscapes, but with the SPADE-COCO in runwayml, there are categories such as:

It is surprisingly good at adding in giraffes where requested.

It also turned my terrible car drawing into something way cooler than I had any right to expect. (I forgot to color in sky, so it appears to have interpreted void as nighttime). 

It didn’t help me out so much with this one, though. It was supposed to be a person standing in front of a brick wall holding a knife. Less gritty than I had imagined.

But the more improbable my requests, the more unpredictable my results. It will fudge my sketched lines more and more to try to have things make sense. 

The clock in the sky tries to anchor itself to the ground. The table has blotches of room beneath it. The toothbrush perched on the potted plant seems to blaze with light. Is the giraffe really wearing boots? The neural net hedges its bets by going a bit blurry.

Here I have clearly requested spheres of giraffe, zebra, and sheep floating above a featureless grassy plain.

It tries to stretch the grass upward toward the spheres, and tries to roughen the tops of the spheres into heads and ears. The whole image is noticeably blocky. 

But with careful choice of materials, I can build scenes like this. Here, um, the clouds are banana, the mountain is cake, the hills are blanket, the grass is teddy bear, the lake is zebra, the house is made of carrot, and the trees are made of wine glasses. It tried its best.

Here I drew a mountain and some water, and then switched water and sky. This neural net actually manages it.

But when I ask NVIDIA’s GauGAN for the same thing, it decides to just submerge the entire scene. I guess this is literally what you’d get. GauGAN is less of a sport about extreme weirdness.

You can play with the runwayml versions of SPADE for free. There are a bunch of other neural nets you can play with as well - runwayml is really an extraordinary tool for people who want to use machine learning creatively without needing to code. An even easier SPADE version to play with is the NVIDIA GauGAN (and a couple of other neat online demos).

Can’t believe how weird/caused this images are in Pix2Pix are with meowth to Alolan Meowth to Galar Meowth.

Fast Pix2Pix - Update

Interesting additions to project by Zaid Alyafeai which implements neural network image translation speedily in-browser. As well as improved training sets for older subjects, you can now doodle with a Pokemon dataset.

Most impressive is the new Faces v2 addition, which employs semantic labelling method to specify certain facial details within your drawing for more accurate results.