TouchDesigner driving Stable Diffusion in real time off a live audio FFT, it dreams to the music

by @kainguyen_ai in c/experimental · · Audio-Reactive

Built a rig where a live audio feed's FFT bands modulate the prompt weights and denoise strength of a fast SD model in real time. Bass hits push the denoise, so the kick drum literally re-dreams the image. Mids drive a set of prompt token weights, highs jitter the seed.

It's not a visualizer bolted onto a video, the image IS being generated by the sound, frame by frame, at about 12fps on a 4090 with a turbo model. Drop the beat and the whole latent space lurches. Sustained notes let it settle into something almost stable before the next hit tears it apart.

Performed it live at a small show last week, generating the visuals to the DJ's set with nobody's pre-render. Terrifying and amazing. TouchDesigner patch and the SD streaming setup in a reply for anyone building similar.

279 points · 8 comments

Comments

  • @ethanpatellab · · 70 points

    real-time SD off a live FFT at a SHOW with no pre-render is the most genuinely experimental thing posted here in months. the risk alone. one dropped frame rate and the whole set dies in front of people.

  • @caseymiller_studio714 · · 48 points

    the no-safety-net part is what makes it art and not a product demo. you're improvising with a model that could faceplant at any second. that's jazz.

  • @elenarodriguesworks306 · · 44 points

    the bass driving denoise strength is such a smart mapping. the kick RE-DREAMING the frame means the rhythm is baked into the generation itself, not slapped on top. how stable is the identity between hits?

  • @kaimiller_art · · 31 points

    it's not, and that's the point, on a sustained section it converges toward something recognizable, then the next kick blows it apart. the music's structure becomes the image's structure. slow songs give you almost-figuration, fast ones give you pure churn.

  • @zarabrown_films · · 19 points

    12fps turbo on a 4090, what model are you streaming, is this SDXL turbo or something lighter

  • @ravichen_creates · · 22 points

    a distilled turbo checkpoint at low steps, 512 res upscaled after. 12fps is the floor before it feels like a slideshow. details and the exact node chain in my top reply.