Advanced: steering motion with a crude grease-pencil guide video as ControlNet input

by @quinnflorespro412 in c/tutorials · · Advanced

For when prompting the motion is not precise enough and you need THIS object to move on THIS path. The technique: make a garbage guide video and use it as a control input.

I literally open a drawing app, scrub through a timeline, and animate a crude colored blob moving along the exact path I want, over however many frames. It looks like a toddler made it. Does not matter.

Feed that as a ControlNet (or Kling's motion-brush / trajectory input, depending on your tool) and the model follows the blob's motion while generating the real content on top. You are essentially keyframing the motion by hand and letting the model handle the rendering.

This is how you get 'the car turns left at exactly the second mark then stops', which no amount of text prompting gives you reliably. It is tedious but it is the difference between hoping and directing. Example guide-vs-output in comments.

80 points · 6 comments

Comments

  • @zaramiller_films6222 · · 19 points

    'you are essentially keyframing the motion by hand and letting the model handle the rendering' is the cleanest explanation of guided video I have seen, this is where it is all heading

  • @avamiller_art · · 12 points

    yeah the future is text for CONTENT, guides for MOTION, nobody is going to text-prompt precise choreography, it is fundamentally the wrong tool

  • @marcusjohnsonworks · · 10 points

    the fact that the guide can look like absolute garbage and still work is the part people do not believe until they try it, the model only reads the MOTION not the quality

  • @adriannascimentolab · · 8 points

    Kling's motion brush is the low-effort version of this for anyone who does not want to make a guide video, you just paint an arrow on the start frame, less precise but 10x faster

  • @dianamartinezlab9040 · · 6 points

    motion brush is great for simple single-object moves, the full guide-video method is for when you need multiple objects on different paths with timing, different tools for different jobs