#audiodiffusion

AI Audio Diffusion: Music Sample Generation w/ Text-to-Audio

Diffusion models are popular right now.

The most noteworthy of these is the recently released Stable Diffusion model, but it appears that this is only the beginning. Indeed, recent work by the Harmonai team has seen diffusion models cross domains, from image generation to audio generation.

In this study, we’ll look at some of the technical underpinnings of diffusion models, first focusing on their history, then on their architecture, and finally on audio creation with the Harmonai colab. Let’s just get started, shall we?

Table of Contents

AI Audio Diffusion: Music Sample Generation w/ Text-to-Audio

A Brief History of Diffusion Models

General Diffusion Model Architecture

Audio Diffusion DIY

A look at Harmonai (+ a surprise)

Dance, Dance Diffusion Revolution

And then… Text-to-image AI for audio?

A Brief History of Diffusion Models

Diffusion models operate by erasing and then retrieving (or noising and then de-noising) the data on which they are trained. Technically, they are inspired by Sohl-Dickstein et alnon-equilibrium .’s thermodynamics. Diffusion models are a subset of Markov random fields (MRFs) in which the Markov chain of diffusion steps gradually introduces noise into the sample data.

The model then learns to reverse the diffusion process, creating new data samples from the noise. The authors show in their 2015 paper, Deep Unsupervised Learning Using Nonequilibrium Thermodynamics, that a model may learn to reverse a diffusion process that perturbs input with noise, resulting in unique data. In addition to these bodies of work, Song et colleagues worked on score-based generative modeling in 2019 that, like diffusion models, disturbed data with several scales of noise.

However, academics did not consider the two topics – score-based generative models and diffusion models – to be more than superficially related at the time. However, in 2020, researchers demonstrated that the evidence lower bound (ELBO), a method used for training diffusion probabilistic models that allows you to rewrite intractable statistical inference problems as tractable optimization problems, is essentially equivalent to the score matching objectives used in score-based generative modeling.

Song and colleagues demonstrated in their ICLR 2021 work that score-based generative models and diffusion probabilistic models are both “discretizations to stochastic differential equations governed by score functions.” Ongoing research in the diffusion space has demonstrated that it has applications not only in image reconstruction (reconstructing medical imagery), but also in a variety of other domains such as molecule generation and defending against adversarial attacks on 3-D point clouds, which could be useful in the autonomous vehicle domain. Let’s look at the architecture that underpins most diffusion models today.

General Diffusion Model Architecture

Audio Diffusion DIY

The application of diffusion models to the domain of audio is a relatively nascent area of research: beginning in the first half of 2021 a body of research emerged regarding diffusion models for:

de-noising text-to-speech

creating probabilistic models for text-to-speech, performing (neural audio) upsampling using a diffusion model

as well as some audio generation diffusion models for generating voice or musical outputs.

The computational complexity of some of these large diffusion models means that training a model from scratch is often out of reach of the home hobbyist, but you can experiment with pre-trained models much like how you can choose various settings for the image diffusion models.

If you’re working on developing an (audio) diffusion model from scratch consider using the U-Net model from the Imagen repository as it’s less resource intensive, but keep in mind the fidelity implications which are discussed on that Imagen repository issue page.

Thankfully, the researchers at Harmonai have released their inferencing repository which allows you to generate various kinds of audio data from scratch using their already-trained diffusion model (inferencing) or you can take an existing piece of music and apply one of several new styles to it:

honk , a style trained on Canada Geese recordings

glitch , an industrial-sounding music style

unlocked, ‘unlocked’ recordings provided by the Internet Archive: a style derived from hundreds of out-of-print LPs across many decades.

and more!

A look at Harmonai (+ a surprise)

Harmonai is a community-driven initiative that distributes open-source generative audio tools to make music composition more accessible and enjoyable for everyone. Simply put, it generates never-before-heard sounds through a process known as diffusion, in which random white noise is refined into sounds based on a pre-trained model that contains millions of parameters calculated during its training process to encapsulate the characteristics of the source material on which it was trained.

The deep learning research lab Harmonai’s mission is to create open-source generative audio models, to assist researchers and developers through computing financing and a community, and to provide new creative tools to artists and audio professionals. Harmonai’s mission is to make music production easier and more pleasant for everyone by developing open-source audio synthesis and modification tools.

Dance, Dance Diffusion Revolution

Actually, just Dance Diffusion.

Consider Stable Diffusion providing just sound rather than visuals.

Dance Diffusion is an open source set of generative music diffusion models. The early models produce 1-3 seconds of audio and can also be used to interpolate and style longer audio recordings.

And then… Text-to-image AI for audio?

On the way!