#chris vik

Kinectar — Chris Vik

Brad and I (Ethno Tekh) are putting together an interactive A/V experience for AUSthetic @ Fed Square (22nd Feb).

We've created a collection of intuitive, expressive and fun audio/visual instruments for the public to enjoy on stage and on the big screen at Fed Square for one night only.

This is the latest performance I've done as Ethno Tekh. The performance took place at Pause Fest's "Stage" event in November 2012. It's a difficult beast to explain, but for those interested I'll give it my best shot....

Our software Ethno Tracker sits at the core of the performance setup. It runs on the PC that is connected to the Kinect camera. It sends the skeleton data from the Kinect over OSC to 2 other computers: one that hosts the custom visual software, the other is runs Kinectar, which I built to turn the movement data from the Kinect into MIDI to control the musical components.

A number of scenes are setup within Ableton Live that synchronize with Kinectar using custom Max4Live patches. The Kinectar project holds a number of "instrument" mappings that directly correlate with the virtual instruments (VSTs) hosted within Live.

Communication from Kinectar to the VSTs is done via internal MIDI, using MIDI Yoke (a really, really old piece of software to allow PCs to send MIDI between software on the same computer).

I choose which instrument I play via a set of MIDI foot pedals on the ground in front of me, and loop the output to layer along-side the drum loops that I also trigger with my foot switches.

Using custom built FFTs built in Max4Live, the audio is analyzed as 3 different layers: Live, Loop and Backing. Live is the audio that's being played by me using the Kinect. Loop is the live stuff that I record and play back as layers. Backing is the pre-recorded loops which I trigger using the foot switches.

This FFT data contains a lot of good stuff that's really useful for generating visual content live - much like a visualizer in most modern media players, only with a lot more data. It's sent via OSC from the audio computer to Brad's visual computer.

Along-side the skeleton information, the FFT data is vital to the success of the audio/visual relationship. The visual software is built in Unity3D and is heavily integrated with Kinectar and the various Max4Live tools used to control the flow of the set.

Just as each audio instrument has it's own sonic characteristics and method of playing it, so too does its corresponding visual component. If you look closely in the video, you'll be able to tell which instrument I'm playing (if not from my movements and the audio) from the visual instrument projected behind me.