#codemusic

So I knew I want to combine my final with Sensing machine class. For the sensing machine class, I made a drawing tool based on humans movement by using Kinect and openFrameworks. I want to make it audio visual

Idea: make an audio/visual interactive work based on viewers movement.

Metaphor: In the visual part, the color will be randomly changed for next color with the movement. For the sound part, I want use some white noise  with a s Markov Chains.

Model: This is a multi-persons playful experience were I want to make everyone happy. Because I can track the depth of the user. It will be a 3D experience as well.

Display: projector for the visual, and a speaker for the sound.

Error: Make the interactive artwork with sound, make 3D audio with visual！

Scenario: Let people play with the randomness.

Task: Standing in the font of the depth camera, than they will start play and explore.

Questions I need to figure out:

Which software I want to use for this project? (Ablation, MAX/MSP, Pd, or Tone.js?)

How do I connect the music data to the kinect, and how the Kinect working together with oF and the sound software?

So for this assignment, I want to play with the idea that drawing as the synthesis.

code:

https://editor.p5js.org/henryiswhy/sketches/S67Bc487Y

My original was that, user can pick a color, and each color have different oscillators, but I couldn’t figure out how to change each oscillator after I change colors. 

So I changed my plan to a simple version of it. With is use mouseX, mousePressed, mouseDragged, mouseReleased to play the sound and drawing. But, it sounds... really bad. haha

Need think about the sound design more.

There is the video when I play with it.

At the beginning, I just want to make a music glove to practice my skills in physical computing and code of music.

So two most important part of this project was how to use the flex sensor and programming nice music in tone.js.   

During the feedback section of the midterm class, Danny asked me a very important question about this project. It’s an interactive work or a work for a performance. 

In the current version, I don’t like the the sound works very well. After play it for a while, it sounds the same. So I need work on the sound design more. And I need a clear concept to push this project in next level. 

I learned a lot in one week. And will be keep working on the love, bring you better version next time.

I am 30% done with my melody project, I will combine this project with my physical computing midterm which I will make a DJ glove that can play music by the flex sensors.

At the current stage, I have some random frequency with keyboard ASDFG, when user play with the keyboard, both the audio and the visual will be changing. And I will add more music elements and a visual system into it by next week.

Interact

https://musiclab.chromeexperiments.com/Kandinsky/

Kandinsky is very different than the other melody interfaces I played with.

How do I do?

I start drawing on the screen naturally. As a user who play with many digital interfaces, when I saw the write canvas, I knew I can play with it.

How do I feel?

I feel more creative in the interface than the other interfaces I played. I spent more time on this one. Because I want to know what's the relationship between drawing and melody. And I think it's a really cool idea as a experiment.

How do I know?

I don't know. The interface provided many elements, draw tools, color and the sound after drawing. I was wondering if color changes the music will change. But I don't think that is true.

https://musiclab.chromeexperiments.com/melody

With designed melody interface, and I like the minimal part of the design.

How do I do?

I checked all the buttons, than I know it's about change the song by change the majors. And the color make sensed for me.

How do I feel?

I feel encouraged, and it's a nice to let the user find out the changes by majors and without showing them. Because I count the numbers for the column as 14, then I found the idea... As someone never play with music, it a good learning experience. And the icon for BPM are so cute!

How do I know?

After play it for few times, and count the numbers for the majors. I knew. And I like the process from I don't know the mean of the column to ohh, it's the majors.

https://learningmusic.ableton.com/make-basslines/good-life.html

A good learning tool for music learner play with rhythms and melodies.

How do I do

I clicked the play button, and listen to the set up song from the system. Then I click the buttons, saw and listen the changes from it.

How do I feel?

I was kind of loss with the set up. Because I don't know I can interact with the interface at the beginning, and later I feel good when I change the buttons.

How do I know?

Listen 

The music I found find melodically engaging is 

I have no experience in music theory, so it's really hard for my determine the mode/scale and draw the melody. After listening the song for around five times, I started drawing the song by  random lines that how me as a listener followed the melody.

I think it  was an interesting experience because it's like drawing the melody base on my own emotion. And the outcome drawing was not bad.

Ideate

Emm, I got a quote very similar to what I did for the listen project. 

"Imagine the music as a moving chain or caterpillar"

Then I start sketch a butterfly, and was think about change the color of the butterfly wing to change the melody of the song. But user can't control the whole progress rate by play with this interface. So I started with the chain as the second idea, and in this idea, user can change the melody by chose cloud (C) to grass (G). I think this is not hard to implement, so I might stay with this idea for the melody code assignment.

Design + Code:

I was really enjoy the beauty of Markov chains especially tone.js have the Tone.CtrlMarkov function which make the algorithm easy to use. So there is not much of the design process, and I was just play with tone.js and p5.js for the project. 

I really want to change the speed of the bouncing ball by the BPM, but I did not make it do work. So now it’s a semi-audio visual drum. Which I should keep iterating.

sketches of ideas:

PingPong

https://fifizhang.com/pingpong

For PingPong the beat and the tempo is very strongly showing once I interactive with it. The interaction part in this project is fun and easy to understand. As a user, when I doing pressed my mouse, their sound and animation will be showing up. And one thing I really like about this project is that once user find the easiest interaction (mouse pressed) is boarding, the designer offered many other play model so the experience can be keep going on. And as a interaction design project, I think the idea of using all the elements in pingpong just so smart and I like it a lot. And the design or the code is not too complex, but it works smoothly in this theme.

AI Duet

https://experiments.withgoogle.com/ai-duet

I think AI duet was a very famous example for algorithmic composition in most recent years. And the idea of a interactive piano was not now. But how might we do it with AI would be a good design question.  In this project, have the elements of rhythm is depend on the user and the neuron network. I found some beauty in this setting. As a person don't really know how to judge a piece of music, I am not sure if when I play keyboard randomly the AI gives me a "good" or a "bad" music. I think this fun part of AI based interactive music is that you can have less user interaction but still make user interactive with the piece by design a good generative music.