using recurrent neural networks for image completion
I read a cool paper today.
https://arxiv.org/abs/1601.06759
It talks about using recurrent neural networks for image generation. Recurrent neural nets basically examine a problem sequentially to predict what something should be based on everything that came before it. They have a lot of uses in problems that have an obvious time domain, like speech recognition, or audio processing, or even things like text generation where the overall context of individual pieces of data plays a larger role.
The interesting thing is that a lot of problems that aren’t obviously sequential in nature can be made into sequential problems and approached with these networks. In this paper they sequentially predict the values of individual pixels based on all the pixels that came before.
Here they show it at work completing images which have had their bottom halves covered up. A lot of the completions look pretty plausible, and show an impressive level of awareness about what’s going on in the image as a whole. One of my favorites is the dogs on the bottom row. All the model has to work with in the occluded image is that tiny wedge of land on the right, so it imagines the dogs sort of floating in a weird misty void. Interestingly it also puts forward a few possibilities that involve the dogs sitting on some surface like grass or sand. It had no way of knowing that from the occluded image, but it knows enough from the data set it was trained on to know that dogs are usually sitting on something.
They also use the model to generate some images from scratch.
These images have an interesting texture. It reminds me of datamoshed images, or the stuff you see when you watch a video in VLC and the decoder completely loses its cool and maps a couch from one scene onto a guy’s face in the next scene. They also have a certain jaggedness to them which other models I’ve seen don’t have.
