Why denoising steps decide what a generated video costs

Ask a text model for a sentence and it writes the sentence once, word by word. Ask a video model for five seconds of a cat on a windowsill, and it draws the whole clip many times. Each draft is a little less noisy than the one before.

That one fact drives much of the GPU cost of video. This note explains it without math: how the work adds up, where the levers are, and why each lever needs a check. It is for anyone who pays for, plans or runs open-source video and image models.

Drawing out of noise

Most open-source image and video models today are diffusion models, or close relatives trained with a method called flow matching. For this post, both work the same way. They start from pure noise, random static in the shape of the output. Then they remove a little noise at a time, guided by your prompt, until only the picture remains. Each removal is one denoising step. The idea dates from a 2015 paper. DDPM showed in 2020 that it can make high-quality images.

Each step is a full pass through a large neural network. The network does not touch only the part of the frame that changes. It reads and rewrites the whole clip, every time.

So the cost of one clip is close to this:

the cost of one pass × the passes in each step × the number of steps

Most steps need one pass. Guidance, which this post explains below, can make it two. Everything else in this post is about one of those three terms.

Multiplier 1: the number of steps

Fewer steps means fewer passes. There are two common ways to get there. A better sampler takes larger steps with the same model, as DPM-Solver shows. Distillation goes further. Engineers train a second model to reach the same result in fewer steps than the original. Progressive distillation is one early, public example of the method.

Distillation has a price. A distilled model is a different model. It can lose detail, light or variety. You have to compare its output with the original before you trust it.

Multiplier 2: the size of each pass

A clip is too large to work on as raw pixels. So modern models work in a smaller latent form. During training, an encoder learns to squeeze clips into that form. To make a new clip, the model denoises in the small space, and a decoder turns the result into frames at the end. Latent diffusion made this approach standard.

Inside the network, the latent clip is cut into small patches across space and time, and each patch becomes a token. Many current video models are transformers that work on these tokens. The DiT paper introduced this design for images.

Here is the important part. Attention is the part of a transformer that lets each token look at every other token. Its compute grows with the square of the number of tokens, while the rest of the pass grows linearly (Attention Is All You Need). If a clip has twice the tokens, attention does about four times the work. So as clips get longer or sharper, attention takes a larger share of each pass. That is why a longer or sharper clip costs much more than a short, small one.

The levers for the size of a pass are:

• Attention kernels. A kernel is a small program that does one calculation on the GPU. An attention kernel that is written for how the GPU’s memory works wastes less time moving data. FlashAttention is a public example.
• Lower precision. Weights and activations can use fewer bits per number. Less data moves, and newer GPUs do more low-precision math per second. The FP8 formats paper describes one common choice.
• Fused operations. Two small steps that each read and write memory can often run as one step that reads once.

Each of these changes the numbers inside the model a little. So each one needs a quality check too.

Multiplier 3: guidance can double each step

Many models use classifier-free guidance to follow the prompt more closely (Ho and Salimans). In its common form, each step runs the network twice. One run uses your prompt and one run does not, and the model moves away from the second toward the first.

That can double the work in every step. Some distilled models need no guidance, which removes the second run. Again, the output can change.

The parts outside the steps

The steps are usually most of the work, but not all of it:

• The decoder. After the last step, the decoder turns the latent clip back into frames. For long, sharp video, the decoder can run out of GPU memory. Decoding the clip in tiles, one part at a time, keeps it inside the memory budget.
• The text encoder. The text encoder turns your prompt into numbers once per clip. It runs once, so it takes little time next to the steps. But its weights can take a lot of GPU memory.
• Loading the model. Weights for a large video model can fill a large part of a GPU. Loading a large model for each request would waste much of the GPU time, so a serving stack keeps models loaded and warm.

What to take away

If you pay for diffusion inference, ask three questions about any setup:

1. How many steps does each output take, and was the model distilled to take fewer?
2. How large is each pass, in frames, resolution and precision, and are the attention kernels matched to the GPU?
3. Is guidance on, and does the product need it?

Each answer points to a lever. Each lever also changes the output. That is why we compare each speed change with the original model’s output. That is our work at Deep Variance. We build the auto-tuning inference stack for production diffusion workloads. It tunes each model for the GPU that runs it and checks each change against the original model.

If you run diffusion models in production and want to compare notes, write to us.