# Special Topics Case Study

# Research & Activity Documentation

# Khant Nyar Lu

# Introduction

I've been producing music for almost four years, so when this course gave me room to explore the intersection of music and interaction design, it felt like the right fit. What I didn't expect was how naturally TouchDesigner would connect to what I already know from Ableton. The node-based workflow, the signal routing, the idea of patching things together: it clicked faster than I thought it would.

Working with Bre made the whole semester easier to navigate. She came in stronger on the visual side; I had more to offer on the audio and technical side. That split turned out to be genuinely useful. We spent a lot of Module 2 asking each other questions across our areas, and by the time we got to the final project, building together made more sense than doing separate things. So instead of one big joint project, we built two smaller collaborative systems where we each led on different parts. One system was more visual, one was more audio. Both of us touched both.

# Module 1: Concept

## Research

Module 1 was mostly about figuring out what we wanted to do. We looked at how TouchDesigner and Ableton get used in live performance contexts: concerts, installations, AV sets. We tried to understand what techniques were actually worth learning. The goal wasn't to find one perfect reference; it was to get a feel for the space so we could start making decisions about direction.

Our design statement came out of that research: explore gesture-based musical interaction through technical work in TouchDesigner, with a focus on technique over a polished final installation. We wanted to build something real and functional rather than demo-y. The idea of learning skills that could eventually transfer to stage AV or interactive exhibits was part of the thinking from pretty early on.

## Action Research Phase

# Project 1: Concept

The concept was straightforward: explore different types of interaction we could layer into a music experience, focusing on actually learning the technique rather than shipping a polished installation. We were both new to parts of this toolchain, so getting things working and understanding why they worked felt more important than aesthetics at this stage.

The bigger picture was always live AV performance: concerts, festivals, raves, museum installs. We kept that in the back of our heads as a way to check whether the skills we were building were worth building.

# Module 2: Prototype

## Research

Module 2 is where I actually started learning TouchDesigner. My approach was basically: find tutorials, replicate them, figure out what I don't understand in the process. It's slower than just watching, but it stuck a lot better. By the end of the module I had three working systems; one music visualizer, one hand tracking demo, one Ableton bridge. That was enough of a feel for the tool to start doing things off-script.

Bre was working through a similar process from the other direction. She focused on getting Ableton basics down and learning MediaPipe hand tracking through the TouchDesigner plugin. By Module 2's end she had two separate hand-tracking systems running with audio and visual components. We compared notes a lot during this phase, which helped both of us move faster than we probably would have working in isolation.

## Action Research Phase 1: Glitch Music Visualizer

## Action Research Phase 2: MediaPipe Hand Tracking + Ableton

# Project 2: Prototype

The prototype phase ended up being two things: a music-reactive visualizer and a gesture-controlled Ableton rig. Neither was finished enough to present as a final project, but both taught me enough about the tools that the final project felt achievable. That was the point.

It also made it clear where Bre and I were strongest. She had a better eye for how the visuals should look and feel. I had more intuition for the audio routing and parameter mapping. That defined how we split the work in Project 3.

# Module 3: Product

## Research

For Module 3, two specific references shaped what we built. For System A, we were looking at @musicazura and @elinvarstudio on Instagram. Both use blob tracking in TouchDesigner as a visual layer over their music. What grabbed me about their work was how physical it felt; your hands are literally revealing something on screen. But looking closer, the audio in their setups doesn't actually respond to the gestures. The music is just playing underneath. That gap felt like an obvious thing to close: same gesture input, driving both the visuals and the sound at the same time.

For System B, the reference was @jasminesokko's performance that went viral. She uses MediaPipe and Ableton to shape her voice in real time through hand gestures. The idea of treating your body as a controller for your own voice, turning a vocal performance into something more like instrument-playing, was what we wanted to explore and build from scratch.

## Action Research Phase 1: System A

## Action Research Phase 2: System B

# Project 3: Product

## System A: Gesture-Driven Audiovisual Filter Experience

System A leaned into Bre's strengths visually, so I told her to build the VFX however she wanted and that I'd figure out the audio around whatever gestures she used. Working reactively like that was actually more interesting than planning everything upfront together.

Bre built the VFX system in TouchDesigner using MediaPipe and a blob tracking workaround. Filters hidden behind a blur layer get revealed through a clarity filter based on where your hand is on screen. I took the same gesture data she was already reading; hand position and pinch, and mapped it to Ableton. Left hand X and Y go to Granulator III's sample position and stereo panning, changing where in the sample the sound plays and where it sits in the stereo field. Right hand pinch goes to stereo width through the Utility device and reverb amount through Echo simultaneously. Open hand, big and wet. Closed hand, mono and dry.

The thing that made System A actually work was that both the visuals and the audio were reading from the same input. Not layered separately; genuinely reacting to the same movement.

## System B: Gesture-Controlled Vocal Performance System

System B was built around using your hands to shape your voice in real time. I handled the Ableton audio system and the TouchDesigner gesture routing. Bre built the visual UI layer that sits on top and gives the performer something to look at while performing.

The Ableton chain is: mic into Vocoder, with a MIDI keyboard providing the carrier tone. This creates a talking synth effect, where the synthesizer takes on the character of your voice. An open palm turns the vocoder on; a closed fist turns it off. After the Vocoder, the signal goes through Auto Pan set to Tremolo mode, where right hand movement controls the vibrato amount.

The performer I was referencing used plugins I didn't have access to, including a third-party tremolo controller. I also deliberately avoided external plugins so Bre could open and test the Ableton file on her own computer without running into compatibility issues. So I recreated everything with native Ableton devices. Auto Pan in Tremolo mode is a pretty decent substitute once you dial in the rate and shape settings.

The TD side follows the same basic architecture as System A: MediaPipe data through select and clamp nodes into the tdAbleton Package. The difference was figuring out which gestures to use. Originally I had hand tilt controlling the vibrato amount, but when we tested it we found that tilting your hand kept accidentally triggering the vocoder toggle at the same time. The gesture data overlapped more than I expected. I switched to tracking hand movement for vibrato instead, which kept the two channels clean.

Bre's UI gives the performer a visual on/off indicator anchored to the palm and a radial pattern that expands with tremolo intensity. It's not just decoration; it tells you what state the system is in while you're performing.

style="width: 90%; border: 5px solid grey; margin-top: 80px" class="w3-round" />

# Research Summation

The research across this semester moved from broad case study work into something hands-on pretty fast. Looking at artists like @musicazura, @elinvarstudio, and @jasminesokko gave us concrete techniques to start from: blob tracking, MediaPipe hand tracking, Ableton bridging. That was more useful than vague inspiration.

The two systems ended up showing two different things gesture-based music interaction can do. System A was about ambient texture: both layers of the experience reacting to the same movement, neither one dragging the other. System B was more performative, closer to instrument-playing, where gestures control something as expressive as your own voice. They're different applications of the same underlying toolchain.

One thing the research reinforced was that working within constraints is useful. Limiting System B to native Ableton devices forced me to actually understand what those devices do instead of relying on something purpose-built. The techniques we learned; MediaPipe in TD, the tdAbleton bridge, gesture-to-parameter mapping, are general enough to apply outside this course context. Live AV rigs, interactive installs, performance tools. That was always the point.

# Reflection

## System A

System A turned out better than I expected. The granular texture from Granulator III has this glitchy, organic quality that actually fits what Bre built visually; it didn't feel like two things forced together. The audio reacts to the gestures in a way that feels musical rather than mechanical, which was what I was going for.

If I could keep working on it, I'd want to get more involved on the visual side. Right now the experience just starts with no context, no onboarding. Adding even a short narrative layer before the interaction begins would make it feel like something you're entering rather than something you're testing. That would take it from a tech demo to something that actually lands as an installation.

## System B

System B took more iteration than I expected. The hardest part wasn't building the Ableton chain or the TD network; it was figuring out which gestures to use and making sure they didn't interfere with each other. My first mapping used hand tilt for vibrato, which seemed clean in theory. In practice, tilting your hand while keeping your palm state stable is harder than it sounds, and the vocoder kept toggling accidentally. Moving to hand movement for vibrato instead fixed it, but it took testing with Bre to catch the problem in the first place.

That was probably the most useful thing about having a collaborator on this system. I wouldn't have caught that issue as quickly testing alone. Gesture-based systems are just hard to evaluate without someone actually performing with them.

## Semester Reflection

I didn't expect TouchDesigner to feel as accessible as it did. I'd heard of it before this course but always assumed it would take months to get anything working. It clicked faster than I thought, and I think that's because I already had a mental model for signal flow from Ableton. Routing data through nodes in TD isn't structurally that different from routing audio through an effects chain. Once I recognized that, I stopped treating it like a completely foreign tool.

That realization was a bit uncomfortable, honestly. I'd been assuming I was a beginner in this space when the foundation was already there. It made me think about what else I might be dismissing before I actually try it.

Working with Bre changed how I think about collaboration too. I usually learn alone; music production is mostly a solo thing, and I'm used to just figuring it out by myself. This semester I found myself moving faster because I had someone to ask. When I was stuck on an audio routing problem, Bre would sometimes reframe it from a visual logic perspective and suddenly the solution was obvious. That's not something I would've gotten from a tutorial. I'm not sure I would've finished Module 3 at the same level without her.

What I'm taking out of this course is mostly the confirmation that the things I care about: music, interaction design, live performance, aren't as separate as I used to treat them. TouchDesigner is the thing that made that concrete. The skills from this semester transfer directly to the kind of work I want to be doing: stage AV, interactive installations, live performance tools. This wasn't a course about software. It was a course about a direction.

x

##

Powered by w3.css