Why every speed-up needs a quality check

Here is an easy way to make a video model faster: make it worse. Cut the steps in half. Drop the precision. Turn off guidance, the second pass that keeps the picture close to your prompt. The model will render faster. It can also draw blurrier faces, flatter light and hands with the wrong number of fingers.

Every speed-up in inference is a trade. Some trades cost nothing you can see. Some cost a lot. The only way to know which is which is to measure. This post is about why we treat that measurement as part of the work, not an extra step at the end.

A faster model can be a different model

The changes that make diffusion inference cheaper all touch the numbers inside the model:

• Lower precision stores and multiplies numbers with fewer bits. Small rounding errors appear in every step.
• Faster attention kernels compute the same thing in a different order, or at lower precision. The results can differ in the last digits.
• Fewer steps, through a distilled model, give a model that learned to imitate the original, not the original itself.

A diffusion model runs its network many times per output, and each step feeds the next. A tiny difference in step one can grow into a different face by the last step. So “it ran faster” tells you nothing about whether it still makes the same video.

Wrong answers can look right

The wider field now measures this. When AI systems write GPU kernels, some of those kernels pass weak tests and fail strong ones. KernelBench-Verified tested AI-written kernels against a stronger baseline and with hidden test inputs. Some kernels skipped work they were meant to do, and the average gain turned into a loss. The Correctness Illusion built a small set of kernels with known bugs. A common test on one input shape passed them. Tests with many random inputs caught every one.

Even correct code can give a user different outputs for the same request. Thinking Machines showed this for language model servers (Defeating nondeterminism in LLM inference). Some kernels give slightly different results at different batch sizes, and the batch size changes with the server’s load. The same is true of every kernel we change, so a test must cover more than one case.

For images and video, verification is harder. Two different videos can both look fine, so “it looks fine” is not a test. With the same prompt and the same random start, the original model gives one reference video. We compare against that.

How we verify a change

Our rule: we compare each speed change with the original model on the same type of GPU. In practice:

1. Render a reference. We run the original model, as its makers ship it, on the same GPU and the same prompts.
2. Render the candidate. We run the changed version with the same prompts and the same seeds, the numbers that set the random start.
3. Compare. We score how close the candidate’s frames are to the reference. We score how well the output still matches the prompt. When people speak, we score whether the words and lip movement hold.
4. Measure speed. We time the candidate, warm, on the same GPU.
5. Decide. We set a quality limit for each model on each type of GPU. A change is accepted only if it is faster and its scores stay above that limit.

Some changes we try fail this. One faster setting moved the frames too far from the original, so it is not our default. Another worked in short tests and broke on long, sharp clips. Those changes do not ship. A record of what failed is as useful as a record of what worked.

Why an autopilot needs this step

Deep Variance builds the auto-tuning inference stack: an inference stack on autopilot, designed for production diffusion workloads. It runs one loop:

1. Profile each model on the GPU that runs it: speed, memory and output.
2. Tune each model to its GPU: precision, attention and memory use.
3. Verify each change with the steps above.
4. Reassess when the model or the GPU changes, and tune again.

A loop that tunes on its own is safe only if it stops changes that make the output worse. The comparison is that stop. Without it, autopilot is only a faster way to make mistakes.

Why it matters for enterprise deployments

When we deploy our stack on a team’s own GPUs, verification is part of the work, not an extra:

• We measure the model on the team’s GPUs and traffic before we change anything.
• We compare each change with the original model’s output on that hardware.
• We record the result, so the team can see what changed and what it cost in quality.

A team that runs a video model in production cannot afford a silent drop in quality. A cheaper output that customers like less is not cheaper. Verification shows, in numbers, what each saving costs in quality, so the team can decide with the facts.

The takeaway

Speed-ups are easy to claim and hard to trust. When anyone offers you faster diffusion inference, ask three questions:

• Faster than what, measured where, and on which GPU?
• Compared with what output, and by which measures?
• Written down where, so you can verify it later?

We ask ourselves the same questions for every change. If you run diffusion models in production and want a deployment you can verify, write to us.