Artist Memo Akten has been exploring methods to generate neural network images in realtime. It started with his #Learningtosee project, with a āblankā neural network, not trained on anything, visualizing fresh visual input:
This is a deep neural network that has not been trained on anything. It starts off completely blank*. It is literally āopening its eyesā for the first time and trying to āunderstandā what it sees. In this case āunderstandingā means trying to find patterns, trying to find regularities in what itās seeing now, and with respect to everything that it has seen so far; so that it can efficiently compress and organise incoming information in context of its past experience. Itās trying to deconstruct the incoming signal, and reconstruct it using features that it has learnt based on what it has already seen ā which at the beginning, is nothing. When the network receives new information that is unfamiliar, or perhaps just from a new angle that it has not yet encountered, itās unable to make sense of that new information. Itās unable to find an internal representation relating it to past experience; its compressor fails to successfully deconstruct and reconstruct. But the network is training in realtime, itās constantly learning, and updating its āfiltersā and āweightsā, to try and *improve its compressor*, to find more efficient internal representations, to build a more āuniversal world-viewā upon which it can hope to reconstruct future experiences. Unfortunately though, the network also āforgetsā. When too much new information comes in, and it doesnāt re-encounter past experiences, it slowly loses those filters and representations required to reconstruct those past experiences. Ā Ā
Following from this, he is using the same realtime method but with a neural net trained on art collections from 150 museums:
#DeepNeuralNet #LearningToSee (havin a hard time cos I'm playin w LR momentum gradclip etc in realtime - no post vfx, all from the DNN)
Here is a link to a video where it re-visualizes a drawing in realtime [EDIT 01/05/17 - Video now uploaded to Vimeo:]
You can follow more updates on the project at his twitter profile here