Built a bot that generates a frame, describes it to itself, then generates the next frame from the description

by @masonsmith_studio898 in c/experimental · · Generative System

Experiment in telephone-as-animation. A vision model looks at frame N, writes a text description of it, and that description becomes the prompt for frame N+1. No image-to-image, only image-to-TEXT-to-image. The visual information has to survive being squeezed through language every single frame.

What happens is fascinating: concrete things that are easy to NAME survive ('a red car,' 'a face'), but anything hard to describe in words dissolves within a few frames. The animation drifts toward whatever the language model finds most describable. Textures die, nameable objects persist and mutate. It's a slow bias toward the lexically convenient.

You're literally watching the bottleneck of language reshape an image over time. The things we have words for survive, the things we don't, don't. Code and a 300-frame example in a reply.

166 points · 8 comments

Comments

  • @dianalee_studio · · 63 points

    this is secretly a profound experiment about language itself. 'the animation drifts toward whatever is describable' means you've built a machine that visually demonstrates linguistic relativity. what we can't name, we lose. genuinely one of the best things posted here all year.

  • @amirmartinez_art · · 47 points

    the Sapir-Whorf render pipeline. the image can only remember what the language can hold. textures have no words so they evaporate, 'a dog' has a word so it mutates but survives. haunting.

  • @zarapatellab4720 · · 42 points

    the fact that nameable objects survive and ineffable stuff dies is such a clean demonstration of the information bottleneck. text is a LOSSY codec for images and you made the loss visible and semantic, not just blurry.

  • @amirgonzalez_art392 · · 29 points

    lossy SEMANTIC compression is the right frame. jpeg loses high frequencies, this loses hard-to-verbalize concepts. same idea, different axis of loss.

  • @emmamiller_vfx · · 22 points

    does it ever hit a stable loop, like a sentence that generates an image that generates the same sentence

  • @novakumar81 · · 31 points

    YES and that's the eeriest part, some runs find a fixed point, 'a woman with dark hair looking at the camera' describes an image that describes the same sentence, and it just locks there forever. a linguistic attractor.