#dcgan

If you’re looking for an image to use as the background of a website or a texture on a 3D model, chances are, you want it to be seamless — if you tile a bunch of instances of the same image as a grid, there won’t be any seams or discontinuities near the edges.

Making this images isn’t easy. But what if we could synthesize repeating images entirely algorithmically, using an existing image as a guide to specify the type of texture we’re looking for.

Gatys et al’s neural style transfer paper suggests that you can synthesize textures with the ordinary neural style transfer algorithm, which I implemented earlier. The trick is to skip specifying a content image input – just ask the network to take a blank canvas and optimize it to match the style of another image.

I modified this to produce seamless images by adding an additional constraint that the boundaries of the image, when tiled side-by-side, also match the style of the source image.

This actually works decently well, and produced a couple of interesting textures:

These images don’t have jarring discontinuities at the boundaries, but they do have some not-so-great quirks — the frequencies of the textures seem to die down near the edges, rather than producing a robust texture that crosses the boundary:

A different approach

I wrote earlier about implementing a convolutional generative adversarial network (GAN) to generate images. These models are good at generating images because they’re trained to distinguish between their own generated images and the “real” images from a dataset, and simultaneously trained to fool this “discriminator.”

I stumbled across a paper describing a variant of GAN called Spatial Generative Adversarial Networks, which claims to be able to generate arbitrarily large textures, as well as repeating textures. It’s like the DCGAN I described before, but with a couple important differences:

- rather than using a random vector as the “seed” for the generator, it takes in a 2D image of random noise, and translates this into a larger, detailed image

- the output of the discriminator is also an image, describing how locally realistic each part of the image is

- there are no fully-connected layers, only convolutions

These changes make it fully convolutional — rather than operating on fixed-size images, it operates by applying convolutional filters across arbitrarily-sized images.

By feeding in arbitrarily large random noise images, you can generate arbitrarily large textures. But here’s the trick: you can make a random noise image, tile it 2x2, feed it into the network, and take the middle 50% of the output image — it’ll be seamless, since both sides were generated from the same noise.

GANs are really hard to train, so it took a long time to find the right hyperparameters for this network — once I did, it actually seemed to work decently well:

🌴 Style transfer-based code

DCGAN training with MNIST data (10 epochs).