#assisted creativity

Example-based Face Stylization - Web Demo

DCGI and Adobe Research have put up an online interactive demo of their stylized facial animation paper.

Just drag and drop an image with a face into it, select one of the styles on the right, hit ‘Submit’ and see what happens ...

Interactive Example Based Terrain Authoring with Conditional Adversarial Networks

Graphics research from LIRIS, Purdue University and Ubisoft is method of generating 3D landscape terrain from simple pen markings with the assistance of neural networks:

Authoring virtual terrains presents a challenge and there is a strong need for authoring tools able to create realistic terrains with simple user-inputs and with high user control. We propose an example-based authoring pipeline that uses a set of terrain synthesizers dedicated to specific tasks. Each terrain synthesizer is a Conditional Generative Adversarial Network trained by using real-world terrains and their sketched counterparts. The training sets are built automatically with a view that the terrain synthesizers learn the generation from features that are easy to sketch. During the authoring process, the artist first creates a rough sketch of the main terrain features, such as rivers, valleys and ridges, and the algorithm automatically synthesizes a terrain corresponding to the sketch using the learned features of the training samples. Moreover, an erosion synthesizer can also generate terrain evolution by erosion at a very low computational cost. Our framework allows for an easy terrain authoring and provides a high level of realism for a minimum sketch cost. We show various examples of terrain synthesis created by experienced as well as inexperienced users who are able to design a vast variety of complex terrains in a very short time. 

Voice Experiments

Project from Experiments with Google is a series of creative explorations of voice recognition technology that run in browser or with smart speakers, ranging from music creation, guesing games and meme making:

Mixlab

Mystery Animal

Meme Buddy

Voice Experiments is a showcase of what’s possible when you bring open-ended, natural conversation into games, music, storytelling, and more. These experiments were made with tools like Dialogflow and the Actions on Google platform, that make it easier for developers to create new voice experiences that work on devices such as the Google Home, or even right in the browser. 

AutoDraw

Assisted Creativity app from Google is a visual drawing tool with autocorrect-like feature, recognizing what your doodle is and suggests relevent pictograms using Machine Learning:

Drawing on your phone or computer can be slow and difficult—so we created AutoDraw, a new web-based tool that pairs machine learning with drawings created by talented artists to help you draw.

It works on your phone, computer, or tablet (and it’s free!). So the next time you want to make a birthday card, party invite or just doodle on your phone, it’ll be as easy and fast as everything else on the web.

Learning What and Where To Draw

Proof of concept computer science research from Scott Reed, Zeynep Akata, Santosh Mohan, Samuel Tenka, Bernt Schiele and Honglak Lee can generate an image based on a selected area with a textual description from neural network trained data:

Generative Adversarial Networks (GANs) have recently demonstrated the capability to synthesize compelling real-world images, such as room interiors, album covers, manga, faces, birds, and flowers.  While existing models can synthesize images based on global constraints such as a class label or caption, they do not provide control over pose or object location. We propose a new model, the Generative Adversarial What-Where Network (GAWWN), that synthesizes images given instructions describing what content to draw in which location.  We show high-quality 128 x 128 image synthesis on the Caltech-UCSD Birds dataset, conditioned on both informal text descriptions and also object location. Our system exposes control over both the bounding box around the bird and its constituent parts. By modeling the conditional distributions over part locations, our system also enables conditioning on arbitrary subsets of parts (e.g. only the beak and tail), yielding an efficient interface for picking part locations.

Flow Machines

Examples of music created with the assistance of Artificial Intelligence from SONY CSL, which are part of an album release due next year.

The more interesting of the two is Mr Shadow, which is based on works by Irving Berlin, Duke Ellington, George Gershwin and Cole Porter, yet has some irregular modern touches:

The second example is based on music by The Beatles:

How is it done? What did the machines do, and what did the artist?

1) We set up a database called LSDB. It contains about 13000 leadsheet from a lot of different styles and composers (mainly jazz and pop about also a lot of Brazilian, Broadway and other music styles).

2) The human composer (in this case Benoît Carré, but we are experimenting with other musicians as well) selected a style and generated a leadsheet (melody + harmony) with a system called FlowComposer. For Daddy’s Car, Carré selected as style “the Beatles” and for Mr. Shadow he selected a style that we call “American songwriters” (which contains songs by composers like Cole Porter, Gershwin, Duke Ellington, etc).

3) With yet another system called Rechord the human musician matched some audio chunks from audio recordings of other songs to the generated leadsheets.

4) Then the human musician finished the production and mixing.