The original paper can be found here (Karras et al., 2022).
Video: https://www.youtube.com/watch?v=T0Qxzf0eaio
This framework reinterprets all diffusion models as solvers for a unified Ordinary Differential Equation (ODE). By decoupling the design choices, the authors significantly improve both training stability and sampling efficiency.
1. A Unified Mathematical Framework
The core contribution is a unified ODE formula that replaces the traditional SDE parameters—drift coefficient $f(t)$ and diffusion coefficient $g(t)$—with two intuitive physical quantities:
- $\sigma(t)$ (Noise Level): The amount of Gaussian noise added at time $t$.
- $s(t)$ (Scaling Factor): The factor by which the clean image is scaled at time $t$.
The Unified ODE Formula
By applying Fourier transforms and convolutional properties, the authors derived the core ODE :
\[dx = \left[ \frac{\dot{s}(t)}{s(t)}x - s(t)^2 \dot{\sigma}(t)\sigma(t) \nabla_x \log p\left(\frac{x}{s(t)}; \sigma(t)\right) \right] dt\]Why this is revolutionary:
- Modularity: You can independently choose $\sigma(t)$ and $s(t)$ based on your needs.
- Straight Trajectories: By setting $s(t)=1$ and $\sigma(t)=t$, the ODE solution becomes a straight line, making it much easier to solve.
2. Training Improvements: Stability & Precision
The authors optimized the training process to ensure the neural network learns more effectively:
-
Preconditioning for Unit Variance: Neural networks struggle with data of extreme magnitudes. The authors designed a scaling schedule to ensure that inputs and outputs are always kept near unit variance, making the loss surface easier to navigate.
-
Informed Noise Distribution: Not all noise levels are equally important for learning. The framework allocates more training capacity to “middle-range” noise—where human-perceivable structures emerge—rather than focusing on extreme noise levels (pure white or pure black).
3. Sampling Improvements: Speed & Quality
Generating images traditionally required hundreds of steps. This paper achieves high-quality results in just a few steps through:
-
Higher-Order Solvers (Heun’s Method): While most models use the simple Euler method, this paper adopts the 2nd-order Heun method. It adds a “correction” step at each iteration. Although it slightly increases computation per step, it drastically reduces the total number of steps required.
-
The “Straight Path” Strategy: By redesigning the noise schedule to be linear ($\sigma(t)=t$), the sampling trajectory becomes a straight line. A straight path is naturally faster and more accurate to navigate than a curved one.
4. Understanding the Scaling: VP vs. VE
The framework clarifies the difference between the two dominant diffusion archetypes by looking at $s(t)$:
| Model Type | Scaling Factor s(t) | Physical Meaning |
|---|---|---|
| VE (Variance Exploding) | $s(t) = 1$ | The original image size is fixed; noise is simply stacked on top. The total energy (variance) “explodes.” |
| VP (Variance Preserving) | $s(t) = 1/\sqrt{\sigma(t)^2 + 1}$ | As noise increases, the image is scaled down (dimmed) to keep the total variance of “scaled image + noise” at approximately 1. |