Anime-style MV where the character sings, cel-shaded look with real lip sync, no rotoscope

by @avaleelab6722 in c/music-videos · · Animated

Anime-style MV where the character sings, cel-shaded look with real lip sync, no rotoscope
Wanted a hand-drawn anime MV feel but with working lip sync to the vocal. Trained a style, kept the character consistent, synced the mouth without rotoscoping frame by frame. Breakdown of the cel-shade pipeline in comments.
youtube.com

Wanted a hand-drawn anime MV feel but with working lip sync to the vocal. Trained a style, kept the character consistent, synced the mouth without rotoscoping frame by frame. Breakdown of the cel-shade pipeline in comments.

374 points · 8 comments

Comments

  • @isabellagonzalez_motion · · 70 points

    the cel-shaded look HOLDING during motion is the impressive part, most AI 'anime' boils and shimmers the second anything moves because it's photoreal cosplaying as 2D. yours has stable flat color fills like actual cel animation. how did you kill the boil?

  • @noahgarcia_vfx · · 52 points

    the boil is the enemy and the fix is a temporal consistency pass plus posterizing the color into flat zones every frame, so the model can't sneak gradient shimmer in. flat fills have nowhere to boil. anime is flat ON PURPOSE and you have to enforce the flatness.

  • @novachopra_vfx159 · · 38 points

    anime lip sync without rotoscope is genuinely new, anime mouths are usually 3 shapes (closed, mid, open) not realistic coarticulation, did you sync to those 3 shapes or go realistic?

  • @marcuspark_fx · · 41 points

    synced to the classic 3-4 anime mouth shapes ON PURPOSE, realistic mouth movement looks WRONG on a cel character. the limited mouth set is the style, so i mapped the audio to the anime phoneme shapes, not to realistic lips. matching the medium's conventions > technical accuracy.

  • @katyamiller_motion · · 28 points

    this is the first AI anime MV i've seen that an actual anime fan wouldn't immediately clock as AI slop. real craft