My full ComfyUI pose-to-video pipeline for dance, with the node graph and why each piece is there

by @sophialee_studio in c/dance-videos · · Tutorial

Wrote this up because the 'just use Viggle' answers in the help threads are leaving beginners stranded the second they want control.

The graph, left to right:
Load driver video → DWPose preprocessor (better than OpenPose for hands/feet, which matter for dance)
Character image → IPAdapter for identity lock
AnimateDiff motion module (v3) as the base animator
ControlNet (pose) fed the DWPose output, strength 0.8 ramping to 0.6 on fast sections
Context options set to 16-frame windows with 4-frame overlap — this is what kills the flicker between batches
Upscale + RIFE frame interpolation to 48fps for the smooth slow-mo look

The settings that actually matter:
DWPose over OpenPose, always, for dance. The foot keypoints are night and day.
Ramp your controlnet strength DOWN on fast motion or you get the stutter-lock where it snaps to each pose like a flipbook.
4-frame context overlap minimum. 2 flickers, 6 gets mushy.

Will answer setup questions below. This is free, go make stuff.

1,187 points · 8 comments

Comments

  • @jordankimworks3156 · · 88 points

    DWPose vs OpenPose for the feet is SO real. switched last month and stopped getting the floor-slide. thank you for putting it in writing.

  • @victordavis_vfx · · 99 points

    The 'ramp controlnet DOWN on fast motion' tip is the thing nobody tells you and it fixed my flipbook problem instantly. Saving this whole post.

  • @kaiwilliams_art · · 93 points

    same, I had it cranked to 1.0 the whole clip because I thought MORE control = better and it was making everything worse

  • @kaidavis_ai · · 54 points

    bless you for writing node graphs out as prose. I can't open someone's .json workflow and learn anything, but THIS I understand

  • @liamlee_creates · · 22 points

    what's your VRAM situation? trying to run this on a 12GB card and the context options are OOMing me

  • @nikolaisinghlab · · 39 points

    drop to 12-frame windows and 3-frame overlap, and switch the base to a pruned fp16 model. you lose a little smoothness but it'll fit in 12GB. the upscale pass you run separately after, not in the same graph, or you'll always OOM.