kayla shomar-corbett @ itp

final project: circular live quantized sequencer

for my final project, i built a circular live quantized sequencer in p5.js using p5.sound. the sketch is based around an 8-step sequencer where each row is mapped to a different sound, and each key on the keyboard places a hit into the pattern.

the main interaction is simple: pressing q w e r a s d f adds sounds to the sequencer, but instead of triggering immediately, the sounds are placed on the next step. this keeps the interaction automatically quantized, so the user can perform with the sequencer without needing perfect timing.

i originally had the sequencer laid out as a rectangular grid, but i changed it into a circular interface so it would feel more like a clock. each ring represents a different sound, and each slice around the circle represents a step in time. this made the visual structure feel more connected to rhythm because the beat literally moves around the circle.

i also added a backing audio loop underneath the sequencer. the triggered sequencer sounds sit slightly quieter than the main track, so they feel like small rhythmic decorations or gestures on top of the song instead of overpowering it. backspace clears the sequencer pattern without restarting the song, which makes it easier to keep performing while changing the texture.

one of the more important visual changes was adding a soft fft-reactive flow field. instead of using fft to draw a literal waveform or a ring of dots, i wanted the sound analysis to create a subtle atmosphere around the sequencer. the sketch analyzes bass, mid, and treble energy, then uses that information to affect tiny pastel particles. when the audio is quieter, the particles move gently. when the audio becomes more active, the particles become brighter, larger, and more animated.

my goal was to make the sequencer feel soft, playful, and performative rather than mechanical. i wanted it to still function like a musical tool, but visually feel more like a small interactive music object.

questions i asked chatgpt while working

how can i turn my rectangular 8-step sequencer into a circular sequencer?

how can i make backspace clear the pattern without restarting the song?

how can i make sure all the sequencer sound files play at the same volume?

how can i line up the q w e r a s d f labels with the first tick so the interface is easier to understand?

why am i getting a getStepAngle is not defined error?

how is fft actually being used in the sketch?

how can i make the fft visuals feel more like a sparse, beautiful flow field instead of dots around the clock?

how can i make the particles respond to bass, mids, and treble?

how can i turn down the sequencer sounds so they sit better underneath the main track?

what i learned

this project helped me understand quantization more clearly because the key press does not directly equal the sound. instead, the key press changes the sequencer’s pattern, and the clock decides when the sound happens. that separation between input and playback is what makes the interaction feel musical and synchronized.

i also got a better sense of how fft can be used indirectly. instead of only visualizing the audio spectrum literally, fft data can control motion, brightness, size, and density. that made the visuals feel more integrated with the sound, while still keeping the sequencer as the main focus.

For this final sketch, I expanded an earlier fullscreen flow-field sketch into an audio-reactive installation piece. The original reference sketch used particles moving through a Perlin noise field, which creates the feeling of invisible currents moving across the screen. I kept that structure, but changed the behavior, visuals, and sound relationship so the sketch felt more personal and more connected to the music.

Each particle samples a noise value based on its x/y position. That noise value is converted into an angle, and the particle moves in that direction. This is what creates the flowing motion.

The sketch uses FFT analysis to read the frequency content of the audio. I split the sound into low, mid, and high frequency ranges. Low frequencies mostly represent kick, bass, and heavier percussion. Mid frequencies capture more of the body of the sound, like chords, melodic material, and voice. High frequencies capture brightness, shimmer, noise, and some drums/sound design.

I experimented with making the sketch behave more like a literal spectrogram, where the y-axis mapped frequency from low to high and the x-axis mapped time history from older to newer FFT frames. I ended up moving away from that because it made the piece feel too diagrammatic. Instead, I returned to a more nonlinear mapping where the whole particle field responds atmospherically to the spectrum.

In the final version, low, mid, and high frequencies affect the entire visual system in different ways. Low frequencies push the particle speed and add brightness. Mid frequencies shift the color and saturation. High frequencies create sparkle, brightness, and more shimmer in the hearts and stars. This makes the sketch feel less like a graph and more like a living field being shaped by the music.

The drums look especially strong because the sketch maps low-frequency energy to particle speed, brightness, and scale. Kick drums and low percussion create spikes in the FFT’s low-frequency range, so when the drum hits, the particles get larger, brighter, and move faster. It makes the percussion feel like it is physically pushing the particle field.

I also changed the particles themselves. Instead of only drawing points, each particle can be a dot, heart, or star. The dots are built with p5’s ellipse(), while the hearts and stars are custom shapes made with vertices and curves. Each particle also has its own size, rotation, and slow spin, so the field feels more varied and less uniform.

For the colors, I switched from simple RGB mapping to HSB color mode. This made it easier to create a more spectral range of colors because hue, saturation, and brightness can be controlled separately. The result is more vivid and reactive than my earlier pastel-only version, but it still keeps the pink, purple, and blue atmosphere I wanted.

The fullscreen button was added to make the sketch easier to present as an installation.

Overall, this piece started as a flow-field particle study and became an audio-reactive visual environment. I wanted it to feel like the music was moving through the particles instead of just playing in the background.

Some questions I asked ChatGPT:

How do I use p5.FFT() to make particles respond to low, mid, and high frequencies separately?

Can you explain how Perlin noise controls the direction of particles in a flow-field sketch?

How can I map bass energy to particle speed in p5.js?

How can I make audio-reactive particles get larger and brighter when the drums hit?

What is the difference between mapping audio to RGB color versus HSB color in p5.js?

How can I make a p5.js sketch use softer pink, purple, and blue colors without losing audio reactivity?

How can I turn point particles into custom shapes like hearts and stars in p5.js?

How do I draw a custom star shape with vertices in p5.js?

How do I draw a heart shape using beginShape() and bezierVertex()?

How can I make each particle rotate or shimmer independently?

Why does loadSound() say it is undefined in p5.js, and how do I fix it?

How can I make an alternate line mode that shows particle motion trails instead of shapes?

How can I make audio reactivity more sensitive without making the visuals too chaotic?

What does it mean to lower the threshold for FFT energy in an audio-reactive sketch?

My p5.js sketch plays audio but the visuals are not changing enough. How can I make the FFT mapping more sensitive?

The drums are making the particles look really good. Why are the shapes getting bigger and brighter on drum hits?

My colors look too similar over time. How can I make the particle colors vary more across the spectrum?

My sketch looks too technical when I map frequency literally. How can I make the audio mapping more atmospheric?

for this assignment i expanded my earlier flow-field particle sketch into a full-screen installation piece that evolves over time and responds to audio.

the original sketch used a flow field built from perlin noise to move particles across the screen. each particle samples the noise field at its position, converts that value into an angle, and moves in that direction. the result is that thousands of particles appear to flow through invisible currents.

in the earlier version the sketch existed mostly as a visual experiment. for this assignment i focused on turning it into something that could function more like an installation or projection piece.

fullscreen behavior

the sketch now runs completely fullscreen using createCanvas(windowWidth, windowHeight) and updates automatically when the browser window changes size using windowResized(). this allows it to adapt to different screen formats such as a laptop display, projector, or large monitor.

audio-driven motion

one major addition was making the system audio reactive. the sketch loads a soundtrack and analyzes it with p5.FFT. the low-frequency energy (roughly the bass range) is mapped to particle speed and visual intensity.

when the bass energy increases, particles move faster through the flow field and the scene becomes more active. when the audio is quieter, the particles slow down and the movement becomes calmer. this creates a tight relationship between the music and the visual system.

section-based visual changes

the visual behavior also changes depending on the position in the song using song.currentTime(). different time ranges trigger different states in the system.

these timed sections change several aspects of the sketch:

the noise seed, which alters the underlying flow field pattern

the color palette, which changes the gradient used for particle color

the light position, which controls which particles are visible

the trail behavior, which affects how strongly previous frames fade

changing the noiseSeed() effectively creates a completely new flow field, so particles suddenly begin moving in different directions while keeping their current positions. this produces abrupt visual shifts that line up with structural changes in the soundtrack.

spotlight system

a moving “light” position determines how visible each particle is. particles closer to the light are drawn with higher alpha values, while particles farther away fade toward transparency.

this creates a spotlight effect where clusters of particles appear to emerge from darkness and then dissolve back into the field.

during one section of the song the light follows the mouse position, allowing some interactive control. in later sections the light jumps to random positions when the bass energy crosses a threshold, creating sudden flashes across the screen.

color mapping

particle color is generated by mapping the particle’s position across the canvas to gradient endpoints.

for example:

the x position influences the red channel

the y position influences the green channel

x again influences the blue channel

because these ranges change between song sections, the same flow pattern can appear with completely different color atmospheres.

particle trails

instead of clearing the screen every frame, the sketch uses a translucent background. this allows previous frames to remain faintly visible and creates long particle trails.

different sections of the piece modify the background alpha and stroke weight so the trails can become smoky, dense, or sharply defined.

installation considerations

to make the sketch behave more like an installation piece, several structural changes were made:

fullscreen responsive canvas

continuous looping audio

automatic visual transitions tied to the soundtrack

optional rendering modes using URL parameters (for example ?mode=lines)

these changes allow the sketch to run continuously without requiring user interaction once the audio begins.

question

This piece is an interactive melody study containing a melody constructed in C Phrygian dominant. This sketch was vibe-coded using ChatGPT, and it was a rushed attempt at a conceptually developed piece without technical application (as it was vibe-coded).

the melody is represented as an array of note objects containing scale degree, time, duration, and velocity. mouse x scales the duration of notes, stretching or compressing the rhythm. mouse y nudges the melody up or down within the phrygian dominant scale, shifting its contour. the reverse transformation flips the order of pitches while preserving the timing structure, creating a retrograde version of the melody.

this sketch represents melody as data rather than fixed notes. the melody is stored as an array of note objects, and each note contains a scale degree, start time, duration, and velocity. instead of storing raw midi pitches, the notes are stored as degrees of a C phrygian dominant scale, which lets the melody stay inside that pitch system when transformations are applied.

the sketch uses Tone.js to synthesize the notes and a simple sequencer advances through the melody step by step. the visualization shows the melodic contour as points on a pitch ladder, with time on the x axis and scale degree on the y axis.

there are a few transformations and interactions. pressing keys can reverse the melody, invert the pitch contour, or transpose it. the mouse also changes the performance in real time: moving the mouse horizontally stretches or compresses the rhythm, while moving it vertically nudges the contour of the melody up or down within the scale.

i cloned the ml5.js repository (git clone https://github.com/ml5js/ml5-library.git) to understand how a machine learning library designed for creative coding is structured under the hood. ml5 sits on top of TensorFlow.js but wraps complex machine learning workflows into simpler JavaScript functions that artists and designers can use in the browser. looking through the repo, the src/ folder contains the core library code, where different model interfaces are defined, while dist/ contains the compiled version that people actually load in their sketches. the examples/ folder was especially helpful because it shows how models like image classifiers or pose detection are typically used in practice with p5.js. one thing that became clear while exploring the code is that most of the heavy computation is handled by pretrained TensorFlow models, while ml5 focuses on simplifying how those models are loaded and how predictions are returned to the user. using Claude to walk through the structure helped me trace how a model gets initialized, how input data is passed into it, and how results are returned to the sketch. this made the library feel less like a black box and more like a set of organized abstractions sitting on top of existing machine learning tools.

i used Claude to help map the structure of the repository and explain how the model-loading process works. the most useful prompts were ones that asked it to walk through the execution flow of the code rather than just summarize the project.

my father is a palestinian refugee from gaza. growing up in the united states, my relationship to that history has always been mediated. mediated through stories, mediated through distance, mediated through documents, borders, and bureaucracies. in some ways, my relationship to palestine is parasocial. i know it through narrative, through inherited memory, through fragments.

for this final project, i wanted to model that mediated relationship using a listening machine.

instead of the computer understanding meaning, emotion, or context, it listens only through the narrow lens of acoustic features and transcription. speech becomes data. words become triggers. meaning collapses into pattern recognition.

the result is an interactive ambient composition titled “tough to talk.”

in the piece, i speak into a microphone. the system listens for specific phrases:

hard to hug

tough to talk

i never fall asleep

these phrases come from marina and the diamonds’ song “starring role.” in the project, fragments of the song appear as sonic responses when the machine detects those phrases.

the system does not understand the emotional weight of what is being said. it simply listens for recognizable patterns and triggers audio events.

this creates a strange interaction. the computer appears to be listening, but it is not listening in the human sense. it is listening computationally. this gap between human meaning and machine interpretation is the overarching theme.

mic

→ python whisper

→ osc messages over udp

→ node bridge converts udp to websocket

→ browser p5 receives osc

→ visuals + transcript + trigger + audio clips

i used OpenAI's Codex and my friend Olivia Brown's computational expertise in the development of this sound art system. when the system runs, the computer continuously listens to the microphone. if speech is detected, whisper transcribes it and sends the result as osc messages. the browser receives these messages and updates the display. if a phrase such as tough to talk is detected, the system triggers an audio fragment from marina and the diamonds’ recording.

the user experiences a conversation with a listening machine. however, the conversation is asymmetrical. the machine cannot understand emotional context or intention. it simply maps speech to predefined triggers.

this project is influenced by readings from the course, particularly jonathan sterne’s work on machine listening. sterne argues that machine listening is not equivalent to human listening. machines do not hear meaning. they hear signals, features, and statistical patterns. the system in this project demonstrates that distinction directly. the computer hears my voice, but it only understands it as a sequence of tokens.

phrases that carry emotional significance in human communication become simple control messages in the system.

for this exercise, i built a voice-controlled p5.js sketch that responds to spoken commands sent over OSC. instead of using keyboard or mouse input, i used my voice as the control surface. i consulted my good friend, Olivia Brown, the example from the notebook, and ChatGPT for troubleshooting and brainstorming.

system architecture

the system is split across three layers:

whisper (python) transcribes live microphone input

python-osc + node bridge sends OSC over UDP → WebSocket

p5.js (browser) listens for OSC messages and updates the visual state

the pipeline looks like this:

mic → whisper → /speech/text                           → /speech/color                           → /speech/number                           → p5.js

whisper transcribes short audio chunks in near real time. the transcription is analyzed for keywords like “pink” or number words like “five.” when detected, python sends structured OSC messages to the browser.

pink as command language

instead of traditional color words like red or blue, i centered the interaction around variations of pink:

“pink”

"pastel pink"

“baby pink”

in python, i mapped spoken color words to RGB arrays and sent them over /speech/color.

in p5, i listened for that address and changed the background:

osc.on("/speech/color", (msg) => { bg = [msg.args[0], msg.args[1], msg.args[2]] })

this let me speak and watch the screen slide through different pink registers in real time.

what felt interesting

language becomes parameter space saying “baby pink” feels different from typing a hex code. the interface becomes semantic rather than numeric.

hallucination and mishearing whisper occasionally mishears. adding RMS thresholding prevented accidental triggers. this made me think about how speech interfaces require filtering before meaning.

i interviewed my father about surveillance.

he is a palestinian refugee from gaza. he is now a naturalized american citizen. i wanted to understand how citizenship reshapes one’s relationship to authority, borders, and visibility.

surveillance is geopolitical.

the passport as a predictive algorithm

when my father traveled with a palestinian passport, he describes:

repeated searches and interrogations in switzerland

prolonged visa registration in egypt

being treated as suspicious before speaking

his passport was not simply a means of identification. it pre-classified him. in contrast, an american passport signals legitimacy. it lowers friction and assumes innocence. a palestinian passport increases friction and assumes risk. surveillance here is bureaucratic. it happens at borders, in databases, in processing times. it sorts bodies before they move.

gaza as a carceral territory

after interviewing my father, i'd describe gaza as an “open-air prison.”

he states that his experience in an american jail allowed more freedoms than living in gaza.

in gaza, he describes:

restricted movement at a territorial scale

constant drone presence with a piercing sound overhead

interrogation and suspicion of hamas activity

witnessing death and detainment

being shot by an idf soldier

the drone is important. rather than being used as just visual surveillance, it emitted sound. it entered the body. it made presence felt at all times. this is ambient domination and subjecting humans to constant surveillance.

erasure as governance

he was unable to:

openly talk about being palestinian

write the word “palestine”

learn palestinian history in school

the curriculum was egyptian. surveillance controls what can be known and remembered. it governs narrative. if simulation is about structuring reality through information, then erasure is a form of enforced simulation.

trauma as the afterlife of surveillance

my father describes persistent ptsd. he lives in fight or flight and struggles to trust authority. surveillance did not end when he left gaza. it became internalized through hypervigilance, distrust, and constant anticipation of threat.

the body remembers what the border forgot.

citizenship as differential exposure

Process + Resources

This sketch started from a Tone.js mixer example we used in class, where multiple audio players are controlled by sliders. I adapted that structure to build a three-stem mixer for my own track, routing each stem through its own signal chain: ping-pong delay for the instrumental, distortion for drums, and a parallel dry/wet reverb bus for vocals.

A key design decision was building the vocal reverb as a split dry/wet path instead of a simple wet knob. I used a constant-power crossfade so the perceived loudness stays stable as the reverb increases, then added a limiter on the vocal bus to prevent spikes. That workflow is inspired by how I’ve handled reverb makeup gain in Reaper and bus limiting in Max/MSP.

Resources that informed this process included:

Class Tone.js player + slider examples

Tone.js documentation (Player, Gain, Reverb, PingPongDelay, Limiter)

Discussion with a friend about routing, gain staging, and fx help

ChatGPT as a troubleshooting/debugging tool when connections or loading behavior broke

The final sketch loads three looping audio stems and treats them like mini mixer channels. Each stem has its own volume slider that controls a gain stage, so you’re basically adjusting channel faders in dB. From there, each stem is routed through a different effect chain: the instrumental goes into a ping-pong delay, the drums go through distortion, and the vocals are split into dry and reverb paths that get blended back together.

The vocal reverb uses a constant-power crossfade instead of a simple wet knob, which keeps the perceived loudness stable as you add more reverb. That blended vocal signal then passes through a limiter so peaks don’t jump unexpectedly. All the sliders simply read their values and smoothly ramp effect parameters, so the system behaves like a small interactive mixer rather than abruptly switching states.

click here for rms analyzer by me (biometric art)

In this sketch, I use RMS (volume data) to determine a labeled “state,” such as agitated, alert, neutral, or quiet. The system captures whatever sound you emit into the microphone and maps volume thresholds to behavioral categories. This turns raw audio into an interpreted state, mimicking how biometric surveillance systems translate signals into judgments about a person.

try this out!

for this assignment, i've curated a collection of short sounds from animal crossing sweet day. using these sounds, i've imported them into a p5 sketch in which i've created a mini rhythmic sequencer. in order to create this sequencer, i used AI to translate my previous Max patches into Java, and i troubleshooted any issues that came up myself.

In double six Again, the user is able to interact with the stems of my song with my collaborator and friend Ben Hochster, the cover art I made for it, a live webcam feed, and a little photo booth strip. The user is able to control the vocal reverb, instrumental panning, and the spacebar-triggered photo booth. My initial goal was to create a tiny interactive album-art distorted photo booth, with the ability to manipulate the sounds you are hearing. The musical photo booth idea came from my experience with social media apps such as TikTok or Instagram.

I've found this experience works best in an enclosed, dimly lit space at the laptop with headphones. The MacBook speakers get the job done just fine, but headphones work great for spatialized pieces, giving my piece a binaural feel.

The user inputs their creativity and curatorial ideas, and the program allows those ideas to materialize. They become part of the artwork through the webcam layer, and the photo strip quietly records a short visual memory of their time inside the sketch.

Nothing in the sketch will start without clicking the screen. This triggers the preloaded stems. When the user clicks anywhere on the canvas, I call userStartAudio() and then start looping all three sound stems (drums, vox, instrumental). The boolean audioStarted flips from false to true, which prevents the stems from accidentally restarting on future clicks (I struggled with this at first). When setup() runs, the webcam is initialized with createCapture(VIDEO) and hidden, since I later draw it manually to control opacity with tint(). This step also creates the UI elements (two sliders), the reverb processing object, and sets up the signal path for the vocal stem by disconnecting it from the main output and routing it through the reverb.

Inside draw(), the algorithm moves into its frame-by-frame logic. The background is drawn first (pastel pink), then the album art covers the canvas. After that, the webcam image is drawn with a 50% tint. The photobooth system is handled through an event/array structure. capture.get() grabs the current video frame and stores it in an array called snaps. If the array grows beyond four images, the oldest one is removed using .shift(). In draw(), a for loop iterates through snaps, placing each thumbnail at the bottom of the screen using the loop index to space them evenly. This creates a constantly updating memory strip of the last four moments the user captures.

As the user interacts, small decisions begin to shape the piece. They can adjust how ethereal or far away the vocals sound by changing the intensity of the reverb mix, or pull the instrumental stem across the stereo field to change its position in the room. Pressing the spacebar lets them capture the moment, leaving a trace behind. At the same time, the strip at the bottom only shows the latest four images, echoing the fleeting, disposable feel of social-media photo filters or photobooth reels.

Even though the sketch is visually simple, it layers several modes of media together: original webart, real-time video, sound design/curation, and interactive controls. Coding it forced me to think not just about how the piece should look and sound, but how each component responds to the user. Instead of a static album cover or a fixed audio mix, double six Again becomes a small, self-contained environment shaped by input, rules, memory, and time.

I've found that when modifying reverb, you are changing the amplitude of the wave which is not measured in decibels. This discovery was alarming to me, as I typically work with decibels in digital audio workstations. The project itself is fairly simple, but being able to modulate each aspect of the listening experience with the addition of music mixing + video/photo ultimately gives the user more choices.

sources:

Reverb sketches: https://editor.p5js.org/p5/sketches/Sound%3A_Reverb

p5 sound reference: https://p5js.org/reference/p5.sound/

Panning reference: https://p5js.org/reference/p5.SoundFile/pan/

For my Physical Computing final, I am inviting you to participate in a small musical photo booth installation. My demo consists of modest materials: cardboard, fabric, a laptop, a breadboard, an Arduino Nano 33 IoT, resistors, wires, three FSR sensors, and green tape. This actualized idea stems from the realm of TikTok, Instagram, VideoStar, and the pervasive digital double.

This project started as a direct product of scrolling through TikTok and envisioning a more analog version. My initial idea was to recreate a TikTok-style “antisocial-social media feed,” where a hardware pushbutton would scroll a p5.js video upward each time you pressed it. I built this by pairing p5’s createCapture(VIDEO) feed with serial input from Arduino, and I actually got the button to scroll the canvas for a brief moment. Even so, adding asynchronous serial communication, layers of video, and real-time image capture made the scrolling unstable (I think). The feed was more “joltable” than “scrollable.” This scrolling development was under the guidance of a good friend, Olivia Brown, and neither of us could figure out how to get it to work.

I decided to simplify the project and turn toward a different but related direction: digital intimacy, play, the digital double, and music listening (before and after the age of the internet). 

The photo booth part is based on p5.js webcam and snapshot examples, where createCapture(VIDEO), capture.get(), and an array of images are used to build a film-strip style interface. The serial communication part follows the structure from NYU ITP’s WebSerial labs, which show how to send multiple sensor readings from Arduino and parse them in p5.js using p5.WebSerial. My main contribution is combining those patterns into an audio stem mixer and photo booth interface that lives inside a soft sculptural installation, where three FSRs and a hidden button let visitors physically remix double six while capturing their own images into the interface.

I initially wanted to make the FSRs into little pillow sensors, but this idea did not come to fruition as I had moved on from the bedroom-creator angle. None of the tutorials I’ve found combine multiple FSRs, a real-time photo booth interface, live audio stem mixing, a sculptural, and fabric and cardboard-based installation with original artwork and music. 

I created the cover art that forms the background of the interface by layering many images with low opacity. The stems came from double six, an original track I produced with my collaborator and friend Ben Hochster, and the entire interface is tuned specifically to the structure and timbre of the piece. The three FSRs act as a physical mixer, where pressing into them changes the volume levels of the drums, vocals, and instruments. The hidden pushbutton triggers the snapshot function, adding your face or gesture into the evolving strip of images.

My goal was to create a soft, intimate platform for solo or group play. Physically modulating the song in real time brings you closer to the artistry itself, bridging the gap between consumer and creator. 

Sources:

Force sensors > arduino > p5.js:

Coding Train’s “photobooth” https://www.youtube.com/watch?v=oLiaUEKsRws + layering overlay https://www.youtube.com/playlist?list=PLRqwX-V7Uu6aKKsDHZdDvN6oCJ2hRY_Ig

Something that inspired me:

More serial communication:

double six | Sound Shrine

click the title!

By Kayla Shomar-Corbett

Influenced by the works of Dan-Ha Lee and Phillip Chao

My final consists of three sound files mimicking an interactive stem-player using Unity. Using 3D objects as tangible digital instruments, the player can trigger stems through keyboard-based interaction. The sound files are vocal, percussion, and instrumental stems from a song I made with my friend and artistic partner, Ben Hochster, titled double six.

During development, I encountered a major setback involving file management. My flash drive was improperly ejected, and my working Unity files were stored on the school laptop. After I hit “build and run,” the game crashed, and I was unable to retrieve the files. 

double six is a remotely-built sound collage that mimics the form and structure of a short electronic pop song. In double six Sound Shrine, the player is limited to a visually dark and aesthetically surreal collage-art gallery. This walkthrough illustrates both the creative and curative process of experiencing music making and listening as a ritual, specifically the relationship between the listener and singer-songwriter music. My private practice consists of multiple recording methods, the overall theme being one-take recordings from my Audio Technica 2020 which are processed via REAPER. My colleague Ben Hochster and I have released multiple ambient works using a recording process similar to “I Am Sitting in a Room,” a 1969 electronic music piece by composer Alvin Lucier utilizing natural reverb through recording multiple tracks on top of each other. In double six, Ben and I have transcended our usual multi-recording process by challenging each other to create an industry-standard pop song. 

My knowledge of Unity as a game design engine is limited, but I have worked with virtual reality in Unity prior to my enrollment in ITP at NYU. In January 2023 at Oberlin Conservatory of Music, I worked on “room.exe,” a VR game where players are situated in a dark and scary bedroom. The players experience the horror by exploring their surroundings, during which the bedroom would slowly morph and cave in, revealing itself to be animate and living. Each interaction with an object triggered a volume increase for different tracks. Using this experience as a launching pad, I was able to apply my 3D object and C# learnings from Immersive Media Design Winter Term at Oberlin to double six – Sound Shrine. 

For this final, I aimed to adapt and refine those earlier tools:

attaching audio sources to 3D objects

mapping keyboard input to object interaction

controlling volume behavior through Unity scripts

staging the environment as an installation space

attaching a camera to the player

Despite the unexpected loss of the original files, the reconstruction allowed me to build a clean and focused version of the piece. I am happy with the final result.

These were my primary sources for understanding how specific components work:

Unity Manual Audiosource

Unity Manual CharacterController

Unity Manual Input System

Unity Manual Collider + Trigger Interaction

Unity Learn Tutorials

Office Hours

room.exe -- Dan Ha Lee + Phillip Chao + I

consult with my web-dev computer science friend Olivia Brown

Over the past few weeks, Antonia and I have been working on a generative composition influenced by Aphex Twin, Björk, Arca, and Radiohead. Although this composition is not very similar, it does utilize digital instruments and percussion.

In P5.js, we implemented a 16-step sequencer that progresses with a timed callback (tick()). Every frame, it checks the current time with millis() and compares it to the previous timestamp. It's like a custom metronome, similar to MaxMSP metro + counter or REAPER's grid quantization. Each step reads from pattern arrays for drums and melody and uses variation tables that were generated at startup. Since we are seeding randomness, the system performs identically every run.

With the help of Chan Jun Shern and my good computer-science-brained friend Olivia Brown, I was able to precompute these arrays and modulate the sound with the use of FX.

I'm extremely familiar with the concepts of scales, rhythm, ADSR, reverb, glitch, panning, delay, and envelopes. The shaping of the envelopes and long-tail reverb transform the oscillating triangle wave, creating space and refined sound. I used A Phrygian dominant because I wanted a darker harmonic sound-- similar to that of the Arab maqam (which I am familiar with due to my Palestinian heritage). The detuning feature adds a more authentic, microtonal aspect. I tried a few detune ranges until it sounded right.

Originally, we randomized everything live, but that was unpredictable, and for the purpose of this assignment, it didn't make sense. I also rewrote the first timing loop using millis(). In the Technology in Music and Related Arts (TIMARA) program at Oberlin Conservatory, I've created Max patches and generative structures before, so the logic was familiar, although the language was different.

The piece ended up sounding more like Brian Eno than Aphex Twin, but I still like it.