#libpd

Due to Android being our first target platform I improved deploying files for this platform. The old workflow consisted of manually copying the pd files to a fixed path on the sdcard and loading them from there. Because the Android app is embedded in a apk file which is a zip file it is not possible to simple fopen the assets content from there. But that is the way pd opens its files. So what I wanted was to simply put files into the StreamingAssets folder in Unity, click build and that is it - o manual copying stuff around.

All the files in the Unity StreamingAsset folder are accessible via simple "fopens" on desktop and iOS but not on Android. One can load the file content via the WWW class but that is no option for pd (fopen). So I need to extract the files from the apk during runtime. On each start the Android version lists all files in the folder StreamingAssets/pd (embedded in the android apk in the assets folder) and extracts it to to Application.persistentDataPath. Now pd can run with plain fopens. This lists the apk content form the StreamingAssets using the AssetManager:

public static ArrayList listAssetsContent(String path) { ArrayList l = new ArrayList(); try { AssetManager am = instance.getResources().getAssets(); // instance is a reference to the activity String [] list = am.list(path); for (String s : list) { String subPath = path + "/" + s; if (subPath.charAt(0) == '/') { subPath = subPath.substring(1); } ArrayList subList = listAssetsContent(subPath); if (subList.size() > 0) { // directory l.addAll(subList); } else { // file l.add(subPath); } } } catch (IOException e) { e.printStackTrace(); } return l; }

And this is how its used (from the kalimba android binding):

private int _openFile (string baseName, string pathName) { // spawn files from path foreach(string path in listAssetsContent(pathName)) { string newFilePath = copyFromStreamingAssetsIntoPersistant(path); if (newFilePath.Substring(newFilePath.Length - 3) == ".pd") { // oggread path hack Regex p = new Regex(@"#X msg ([0-9]+) ([0-9]+) open (.*[/\\])?([^/\\;]+);", RegexOptions.Multiline); string t = File.ReadAllText(newFilePath); t = p.Replace(t, "#X msg $1 $2 open " + Application.persistentDataPath + "/pd/$4;"); File.WriteAllText(newFilePath, t); } } string file = Application.persistentDataPath + "/" + pathName + "/" + baseName; AndroidJavaClass jc = new AndroidJavaClass("org.puredata.core.PdBase"); return jc.CallStatic("openPatch", file); } // based on something found on stackoverflow public string copyFromStreamingAssetsIntoPersistant(string p) { string filepath = Application.persistentDataPath + "/" + p; // this is the path to your StreamingAssets in android WWW f = new WWW ("jar:file://" + Application.dataPath + "!/assets/" + p); // CAREFUL here, for safety reasons you shouldn't let this while loop unattended, place a timer and error check while (!f.isDone) { } // then save to Application.persistentDataPath var fileDirectoryParts = filepath.Split('/'); if (fileDirectoryParts.Length > 1) { string directory = string.Join("/", fileDirectoryParts.Take(fileDirectoryParts.Length - 1).ToArray()); Directory.CreateDirectory(directory); } File.WriteAllBytes (filepath, f.bytes); return filepath; }

Currently I copy everything everytime a patch file gets opened. Not perfect but perfectly simple :). As long as the pd files and dependencies (wav, ogg) stay small it is not worth the effort to implement a complex version-checking-system that only extracts new or changed files. But perhaps this will become necessary in the future. As one can see there is a ugly "Regex p = new Regex(@"#X msg ([0-..." kind of hack in the _openFile method.  We encountered some problems with paths using the oggread module (that we recently integrated). oggread does not respect pd's search paths and simply opens ogg files from the current working directory or by absolute path. A consistent working directory on every platform including our composer's workstation was nearly impossible :) so we went for the absolute path way. Fillipo (our composer) opens the ogg files with a absolute path (oggread's open message) to StreamingAssets/pd and put the ogg file there. On the device the _openFile method searches for open messages "#X msg ([0-9]+) ([0-9]+) open (.*[/\\])?([^/\\;]+);" in the pd files and strips the absolute path. "/my/lala/music.ogg" turns into "music.ogg". Not very elegant but at least no manual adjusting/copying stuff. I also integrated the seq module from cyclone for parsing midi files. Before this Filippo converted the midi files using pd-extended into some text representation and copy & pastes this text into the pd files used on the device. So again less manually copying stuff ;) yeah.

"Pure Data (Pd) is a visual programming language developed by Miller Puckette in the 1990s for creating interactive computer music and multimedia works." http://en.wikipedia.org/wiki/Pure_Data

Since Tridek will use PD (Pure Data) for audio generation someone needs to glue together the existing pd code and the Unity engine. That someone is me, Sebastian (aka hagish), software developer at the Bit Barons.

Here's what I have implemented so far. If you're interested in the full source code: We plan to release the whole Unity-libpd package as an open source Unity plugin (called Project Kalimba) at a later point.

Content of this post

API to communicate with Pd

Using Pd in Unity in edit mode

Libpd on iOS with API bindings

Libpd on Android with API bindings

Problems and future prospects

API to communicate with Pd Using Pd the game core can focus on sending game centric events and data (like players health) to the audio engine (Pd) and Pd transforms this into audio. That is the API (C#) we created for this communication direction (canonical mapping to libpd API):

// triggering stuff void SendBangToReceiver(string receiverName); // sending data void SendFloat(float val, string receiverName); void SendSymbol(string symbol, string receiverName); // obviously there need to be a way to initialise everything void Init();

// loading and closing patch files within pd void CloseFile(int patchId) int OpenFile(string baseName, string pathName);

Pd itself is also capable of sending events back but we do not need this at the moment. Therefore we skipped the "back direction".

Using Pd in Unity in edit mode For testing Pd in edit mode we decided to connect Unity via tcp to a local vanilla Pd instance. So there is no internal libpd in the edit mode. This way one can adjust the patches in the Pd gui at runtime without restarting the game. Also debugging the patches is a lot easier. The downside is that OpenFile/CloseFile does not work automatically. One needs to manually load the requested patch files into Pd. Unity play screen (left) with some test buttons that trigger some output (buttom) in a running pd patch (right).

In our example Unity connects via localhost to port 32000 and sends a bang to the receiver "test2". To receive these messages in Pd you need to add a "netreceive 32000" object with a linked empty message box (see (2) in the screenshot). In Unity the message "test2 bang" gets transformed into

set; addsemi; add test2 bang; bang;

to trigger a "; test2 bang" (sends a bang to the object test2) message in the connected message box. This way one can address objects in Pd via receiver name without extra objects/links/messages in the pd patch. Just one netreceive + message. Here is a C# snippet from the implementation.

public override void SendBangToReceiver(string receiverName) {     setup();     constructAndSendMessagesToSendMessage(receiverName + " bang"); } private void constructAndSendMessagesToSendMessage(string message) {     sendPdMessage("set;");     sendPdMessage("addsemi;");     sendPdMessage("add " + message);     sendPdMessage("bang;"); } // no need adding a closing ; private void sendPdMessage(string message) {     if (client != null && client.Connected && stream != null)     {         byte[] ba = asciiEncoding.GetBytes(message.Trim().TrimEnd(new char[]{';'}).Trim() + ";");             stream.Write(ba, 0, ba.Length);     }     else     {         Debug.LogError("could not send message " + message + " to " + client);     } }

Sending data and other stuff works analog.

Libpd on iOS with API bindings The libpd implementation works straightforward with pd's iOS binding. Only some small adjustments to the xcode project generated by Unity were necessary: * additional build flags

-mno-thumb -DHAVE_UNISTD_H -DHAVE_LIBDL -DPD

* in AppController.h

#import "PdAudioController.h" #import "PdBase.h" @property (nonatomic, retain) PdAudioController *audioController;

* in AppController.mm after startUnity

self.audioController = [[[PdAudioController alloc] init] autorelease]; [self.audioController configurePlaybackWithSampleRate:44100 numberChannels:2 inputEnabled:YES mixingEnabled:NO]; [self.audioController configureTicksPerBuffer:128]; [PdBase openFile:@"main.pd" path:[[NSBundle mainBundle] resourcePath]]; [self.audioController setActive:YES]; [self.audioController print];

* in AppController.mm at dealloc

self.audioController = nil;

* in AppController.mm after @implementation AppController

@synthesize audioController = audioController_;

The workflow for building patches and necessary resources into the IPA is currently quite provisional. One needs to manually add all necessary files into the xcode project as resources.

Libpd on Android with API bindings On Android we use Pd's Android binding with a custom Unity player activity (for setup and teardown):

... public class CustomUnityPlayerActivity extends UnityPlayerActivity {     private static final int SAMPLE_RATE = 44100; ...        @Override     protected void onCreate(Bundle savedInstanceState) {         super.onCreate(savedInstanceState);         try {             initPd();         } catch (IOException e) {             toast(e.toString());             finish();         }     }     @Override     protected void onStart() {         super.onStart();         PdAudio.startAudio(this);     }     @Override     protected void onStop() {         PdAudio.stopAudio();         super.onStop();     }         @Override     protected void onDestroy() {         cleanup();         super.onDestroy();     }         private void initPd() throws IOException {         if (AudioParameters.suggestSampleRate() < SAMPLE_RATE) {             throw new IOException("required sample rate not available");         }         int nOut = Math.min(AudioParameters.suggestOutputChannels(), 2);         if (nOut == 0) {             throw new IOException("audio output not available");         }         PdAudio.initAudio(SAMPLE_RATE, 0, nOut, 128, true);     }     private void cleanup() {         // make sure to release all resources         PdAudio.release();         PdBase.release();     } }

The binding (SendBangToReceiver, ...) is straightforward calling pd Android from C#:

private void _sendBangToReceiver (string receiverName) {     AndroidJavaClass jc = new AndroidJavaClass("org.puredata.core.PdBase");     jc.CallStatic<int>("sendBang", receiverName); }     private void _sendFloat (float val, string receiverName) {     AndroidJavaClass jc = new AndroidJavaClass("org.puredata.core.PdBase");     jc.CallStatic<int>("sendFloat", receiverName, val); } private int _openFile (string baseName, string pathName) {     AndroidJavaClass jc = new AndroidJavaClass("org.puredata.core.PdBase");     // TODO this is just provisional     return jc.CallStatic<int>("openPatch", "/mnt/sdcard/kalimba/" + baseName); }

As you can see in the _openFile implementation there is a hardcoded path to the sd card. Building pd patches and resources into a apk is currently not implemented. One must copy them manually onto the sd card.

Future prospects

As you can see, this is a very short introduction on how to integrate libpd with Unity on mobile devices. It represents exactly the work we've already done so far. Over the course of Tridek's development, I'll have to fix a few more problems and expand the current implementation. This is why this post is only the first part of the whole series.

Here's what's still to-do:

The build process is completely provisional. On both platforms one need to manually copy patches and resources. We plan to improve this in the future ;)

Performance-wise Pd works better on iOS than on Android. But this is based on non-scientific observations only. In our small test programs both platforms work good enough.

The pd -> game communication direction is missing. We are not sure if we will need it. So currently there is no plan to implement it.

We plan to add the OGG and MIDI file reader objects. Currently there is only plain vanilla pd.

We plan to release Kalimba as an open source Unity plugin.

More information / links

http://puredata.info/downloads/pure-data - vanilla pd client

https://github.com/libpd/libpd - libpd

https://github.com/libpd/pd-for-ios - libpd ios binding

https://github.com/libpd/pd-for-android - libpd android binding

http://createdigitalmusic.com/2012/07/android-high-performance-audio-in-4-1-and-what-it-means-plus-libpd-goodness-today/

We’re envisioning an electronic score for Tridek that can also incorporate elements of the savage, raw nature of the planet in form of acoustic percussive elements. The overall genre though, if we had to pick one, will be techno/downtempo style, depending on the game situation. Tridek being a trading card game, we need these two paces in order to accommodate for longer playing sessions, and hey, I’m a huge fan of everything downtempo anyways (think Portishead, Massive Attack, Kruger & Dorfmeister, Röyksopp and the like)!

The percussive elements can be sampled, and pretty “dirty” in my opinion. I love the tension that’s created between hyperpolished synths and raw samples. With PD (see below) we can even have some melodies play and adapt to the game situation in real time.

Will there be a central musical theme in Tridek? We don’t know yet. It depends how the characters and the universe will evolve. We certainly won’t force a thematic structure on a game that may not need it. The shift to a more electronic style is very welcome on my part, since my last projects and most current ones have a very acoustic-centric sound aesthetic. I love the idea of tinkling with more synthesizers and creating beats, it’s been a while! Plus, our artist Alex Widl is himself a talented Trance music producer, so we’ll be exploiting this synergy as well.

The core of it all: Puredata

We’ve decided to use Puredata, an open-source visual programming language originally intended for composers and visual artists, as the audio engine for Tridek. This means that it will run in parallel with Unity on mobile platforms. Our resident programmer Sebi managed to interface Unity and PD on the Prototyping level, and PD is now running on Android and iOS as well, using libpd.

Why Puredata, though? At first, it seems cumbersome to do all this prep-work for the audio part. Interfacing PD, making sure the programs can communicate between each other, porting the whole thing to mobile, etc. etc. why? The answers are many. With PD, I can singlehandedly “program” my own audio engine. I can construct synthesizers that play music in real time in the game. I can add effects, use samples, build interactive music structures…you name it. It’s the (geeky) game composer’s dream come true. Especially for a synth-based score, we can expand the interactivity of Tridek’s music and sound effects dramatically by using PD.

Of course, all the usual suspects to programming apply in this case, too: real time synths consume processing power, so we’ll have to keep it all nice and lean, and we’re still experimenting with what will be just “too much”. Our initial tests make us optimistic, though, since PD via libpd is very compact and stable. Nice!

I’ve recently played around recreating the NES sound chip in Puredata and porting it to mobile, and it works just nice. It’s under 100k, so you could make a whole chiptune soundtrack with this thing using almost no space at all, let alone samples of any sort!

Using PD every day it becomes clear that really, anything is possible as long as you know how to build it. I’m very glad I can also take away some work from our programmers, as I know they have enough to deal with anyways, and frankly the last thing you want is a crazed composer coming to you asking for some ridiculous audio gadget! ?

In the next weeks, we’ll be experimenting with different applications of interactive audio structures. Maybe we’ll be throwing in a bit of generative music too…anything that complements the upcoming game well. Stay tuned.

On a more philosophical note, I firmly believe that the future of game composers lies in taking real control over the technical aspect of game music structure, DSP and all that good stuff. Not just to show off that you can do more than putting notes together, but to perfectly blend your composition with the game- which needs, as we all know, very specific considerations compared to all other forms of music nowadays. This is what makes composing for games so challenging and exciting!