The Problem With Moving People
Generating a static image of a person is a solved problem. Stable Diffusion does it brilliantly. But ask it to recreate a specific pose — say, a dancer mid-leap — and things get complicated fast.
This was the core challenge behind Magic Dance: given a reference video of person A dancing, generate the same motion performed convincingly by person B.
How ControlNet Changes the Equation
ControlNet is a neural network architecture that conditions diffusion model generation on spatial control signals. For human motion, that control signal is a pose skeleton — typically extracted using a pose estimation model like OpenPose or DWPose.
The pipeline looks like this:
Input Video → Pose Extraction → Skeleton Sequence
↓
Target Person Image → ControlNet Conditioned Diffusion → Output Video
The magic is that ControlNet preserves the geometry of movement while diffusion handles the appearance of the target person.
What I Got Wrong First
My initial assumption was that frame-by-frame generation would work. It doesn't.
Each frame generated independently leads to temporal inconsistency — flickering, identity drift, clothing that changes between frames. The model doesn't "know" it's generating a sequence; each frame is an independent image to it.
The solution was to use temporal attention layers — modifications to the UNet architecture that allow the model to attend across frames during generation. This dramatically improves consistency.
The Technical Reality
Diffusion models are expensive. Generating a 3-second clip at 24fps — 72 frames — took around 8 minutes on a single A100. For a research project, that's fine. For production, you'd need:
- Aggressive batching
- Frame interpolation to reduce generated frames
- Distilled/faster diffusion variants (like LCM)
What Surprised Me
ControlNet's understanding of the human body is remarkably semantic. When I gave it anatomically impossible poses (testing edge cases), it would "correct" them — smoothing out angles that real joints can't achieve. The model has learned human biomechanics implicitly from training data.
That's the part that stays with me. These models have built internal representations of the world that we didn't explicitly program. Magic Dance worked not just because of careful engineering, but because the underlying models understand bodies in a way that emerged from scale.
Interested in the technical details? The code is on GitHub. Happy to discuss the architecture.