#cyclegan

CycleGAN feedback loop

Visual experiment by @mario-klingemann resembles a combination of DeepDream and cellular automata, generating abstract patterns with facial features:

Two CycleGAN models trained on face transformations are passing their outputs back and forth creating an infinite loop of ever-changing Turing patterns. 

CycleGAN two model feedback loop

See, a thing I get really excited about with neural nets (and other forms of procgen) is when they output something that we couldn’t have gotten any other way. No matter how bizzare the output, what we’re seeing is something new and alien.

Lately, Mario Klingemann has been experimenting with feeding neural nets (Mostly CycleGAN) into each other. The published results have that alien quality I value: mostly coherent but in a way that no human would have chosen, let alone manually drawn.

Two uses of generative art: bringing to life something that I can picture but would take far too long to do by hand, and creating something I that I would never have been able to imagine on my own.

There’s some overlap here: making a convincing forest, for example, requires a lot of little details that would take a lot of effort for me to imagine by hand. Like most humans I am bad at randomization, and the little details of moss and undergrowth can sometimes be done more effectively with a good generator that mimics the natural generation. But those aren’t quite as abruptly startling as the images that are literally impossible for me to have pictured before I saw them.

These videos are examples of both of these uses: the process would have taken far too long to do by hand, and the nature of the process is one that I wouldn’t have come up with without first seeing it here.

Of course, as we become familiar with the imagery it becomes part of our thinking. Maybe not the exact process--humans are prone to pattern recognition, after all, so I’m likely to end up with a shortcut imitation of the process unless I study it closely. (A ton of formal fine art training is just learning to study things closely, to understand the original processes and develop better shortcuts.)

CycleGAN

One big drawback of previous style-transfer methods was that you needed to train the network on image pairs. In order to figure out the similarities you’d need something like a photo and a painting of a photo. Unfortunately, there aren’t many examples of that in the wild. Things like semantic annotations helped, and there have been attempts with automated processes, but this was a general limitation.

As you might guess, that’s not true anymore. Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A. Efros,, the same team who brought us pix2pix have come up with a way to do the training without paired images.

The results are, well, pretty good:

Remember, the one of the limitations of things like edges2cats is that the edges were created with an automated process that missed a lot of details or highlighted irrelevant ones. Being able to use completely separate datasets for the training opens up a host of new options.

https://arxiv.org/abs/1703.10593

GAN, DCGAN, CGAN, Pix2Pixに引き続きGAN手法のお勉強。 https://blog.negativemind.com/2019/12/29/pix2pix-image-to-image-translation-with-conditional-adversarial-networks/ Pix2Pixからだいぶ時間が空いてしまったけど、次はCycleGANについて。

CycleGAN

CycleGANはICCV 2017で発表された論文 Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networksで提案されたGANによる画像変換手法。 馬の画像(動画)をシマウマに変換したこちらの衝撃的な動画が有名ですね↓