TouchDesigner driving Stable Diffusion in real time off a live audio FFT, it dreams to the music
Built a rig where a live audio feed's FFT bands modulate the prompt weights and denoise strength of a fast SD model in real time. Bass hits push the denoise, so the kick drum literally re-dreams the image. Mids drive a set of prompt token weights, highs jitter the seed.
It's not a visualizer bolted onto a video, the image IS being generated by the sound, frame by frame, at about 12fps on a 4090 with a turbo model. Drop the beat and the whole latent space lurches. Sustained notes let it settle into something almost stable before the next hit tears it apart.
Performed it live at a small show last week, generating the visuals to the DJ's set with nobody's pre-render. Terrifying and amazing. TouchDesigner patch and the SD streaming setup in a reply for anyone building similar.