Mathematical Principles of Stable Diffusion

1. Score Function and Score Matching

Once the score function is known, one can sample from $p(\bx)$ via Langevin dynamics:

In the limit of step size $\eta \to 0$ and number of iterations $K \to \infty$, $\bx_K \sim p(\bx)$.

Score matching

We approximate the score function with a neural network $\boldsymbol{s}_\theta(\bx)$. The direct score-matching loss is:

Because the true score is unknown, we use denoising score matching (Vincent, 2011). For the distribution $p_\sigma(\tilde{\bx}|\bx) = \N(\tilde{\bx}; \bx, \sigma^2 I)$ obtained by adding noise $\sigma$:

For Gaussian noise the conditional score can be computed explicitly:

2. A Unified Formulation via SDEs

Song et al. (2021) gave a unified formulation of diffusion models as continuous-time stochastic differential equations (SDEs).

VP-SDE (variance preserving)

The SDE corresponding to DDPM takes the following form:

Here $\beta(t)$ is the noise schedule, and $\beta(t) = \beta_{\min} + (\beta_{\max} - \beta_{\min})t$ corresponds to a linear schedule.

VE-SDE (variance exploding)

This corresponds to SMLD (Score Matching with Langevin Dynamics):

Probability-flow ODE

For any SDE there exists an ordinary differential equation (ODE) that produces the same marginal distributions $p_t(\bx)$:

3. Derivation of the DDPM Variational Bound

The DDPM training objective is derived from the variational lower bound (ELBO) on the data log-likelihood.

Lower bound on the log-likelihood

The variational bound $\L_{\text{VLB}}$ decomposes into three types of terms:

Computing the posterior

By Bayes' theorem, $q(\bx_{t-1}|\bx_t, \bx_0)$ can be computed in closed form as a Gaussian:

where:

Reparameterization and the simplified loss

Using $\bx_t = \sqrt{\bar{\alpha}_t}\bx_0 + \sqrt{1-\bar{\alpha}_t}\bepsilon$ to eliminate $\bx_0$ and express the mean in terms of $\bepsilon$:

Designing the mean of $p_\theta$ as $\bmu_\theta(\bx_t, t) = \dfrac{1}{\sqrt{\alpha_t}}\left(\bx_t - \dfrac{\beta_t}{\sqrt{1-\bar{\alpha}_t}}\bepsilon_\theta(\bx_t, t)\right)$ gives:

4. The ODE Interpretation of DDIM

Song et al. (2020) proposed DDIM, which uses the same trained model $\bepsilon_\theta$ as DDPM while constructing a non-Markovian reverse process.

Because DDIM ($\sigma_t = 0$) corresponds to a discretization of the probability-flow ODE, computing it over a subset of timesteps $\{\tau_1, \tau_2, \ldots, \tau_S\} \subset \{1, \ldots, T\}$ ($S \ll T$) yields good samples even with a small number of steps.

5. VAE Latent Space and Latent Diffusion

The core idea of Stable Diffusion is to perform diffusion not in pixel space but in the VAE latent space (Rombach et al., 2022).

The math of the VAE

An image $\bx \in \R^{H \times W \times 3}$ is compressed by the encoder $\mathcal{E}$ into a latent $\bz = \mathcal{E}(\bx) \in \R^{h \times w \times c}$ and reconstructed by the decoder $\mathcal{D}$. The loss is:

Stable Diffusion uses $f = H/h = W/w = 8$ (8× downsampling) and $c = 4$. The VAE is pretrained independently of the diffusion model.

Diffusion in latent space

Rather than the pixel-space $\bx$, the usual diffusion (noising/denoising) is applied to the latent $\bz_0 = \mathcal{E}(\bx_0)$:

At generation time, after obtaining $\bz_0$ from $\bz_T \sim \N(0, I)$ by reverse diffusion, we map back to pixel space via $\hat{\bx} = \mathcal{D}(\bz_0)$.

6. The Math of Text Conditioning

The text prompt is converted by the CLIP text encoder into a sequence of feature vectors $\boldsymbol{c} = [\boldsymbol{c}_1, \ldots, \boldsymbol{c}_L] \in \R^{L \times d_c}$. This condition is injected via Cross-Attention layers inside the U-Net:

Here $\varphi_i(\bz_t)$ is an intermediate representation of the U-Net (the flattened feature map of layer $i$). Queries come from the image features, while Keys and Values come from the text features. In this way the semantic information of the text spatially guides image generation.

7. Derivation of Classifier-Free Guidance

We give the mathematical basis of Classifier-Free Guidance (CFG; Ho & Salimans, 2022), which controls the quality of conditional generation.

Derivation from Bayes' theorem

The score of the conditional distribution is:

We amplify the score with a guidance strength $w$:

Substituting $\nabla_{\bz}\log p(c|\bz) = \nabla_{\bz}\log p(\bz|c) - \nabla_{\bz}\log p(\bz)$:

During training, with a fixed probability (typically 10%) the text condition $c$ is replaced by the empty condition $\varnothing$ via a random drop, so that a single model learns both conditional and unconditional generation.

8. The Math of the Noise Scheduler

The choice of noise schedule $\{\beta_t\}_{t=1}^T$ greatly affects generation quality.

Linear schedule (DDPM)

Cosine schedule (Nichol & Dhariwal, 2021)

Designed so that $\bar{\alpha}_t$ decreases uniformly:

$s = 0.008$ is an offset (it prevents $\beta_t$ from becoming too small near $t = 0$).

Compared with the linear schedule, the cosine schedule distributes the signal-to-noise ratio (SNR) more uniformly across intermediate steps, improving quality especially for high-resolution images.

9. v-prediction (SD 2.x)

SD 1.x predicts $\bepsilon$ (the noise), whereas SD 2.x adopted v-prediction (Salimans & Ho, 2022). The prediction target $\boldsymbol{v}$ is defined as a linear combination of $\bx_0$ and $\bepsilon$:

Advantages of v-prediction

• SNR-independent stability: with $\bepsilon$-prediction, as $t \to 0$ (high SNR) the signal dominates and the gradient can become unstable. $\boldsymbol{v}$ corresponds to an angular parameter between noise and signal and yields a stable gradient regardless of $t$.
• Numerical stability: in the limit $\text{SNR}(t) \to \infty$, the $\bepsilon$-prediction loss diverges, while the $\boldsymbol{v}$-prediction loss stays bounded.

The loss function simply replaces the prediction target:

OpenCLIP ViT-H/14 text encoder

SD 1.x used OpenAI's CLIP ViT-L/14 (768 dimensions, about 123M parameters), but SD 2.x switched to OpenCLIP ViT-H/14 (1024 dimensions, about 354M parameters).

• Increased output dimension: the Key/Value of Cross-Attention go from 768d to 1024d, giving a richer semantic text representation. The sizes of the U-Net's $W_K, W_V$ projection matrices also change to $d_{\text{model}} \times 1024$.
• Trained on LAION-2B: OpenCLIP is trained on the open dataset LAION-2B, removing the dependence on OpenAI CLIP's private dataset.
• Penultimate layer: SD 2.x uses the output of the second-to-last (penultimate) layer of the text encoder rather than the final layer. The final layer is too specialized for the contrastive learning objective, whereas the intermediate layer has richer features for the generation task.

768×768 default resolution

In SD 2.1 the training resolution was raised from 512×512 to 768×768. In latent space this becomes $64 \times 64 \to 96 \times 96$ (the downsampling factor $f = 8$ is unchanged), and the number of tokens the U-Net processes increases from $64^2 = 4{,}096$ to $96^2 = 9{,}216$. Since the cost of Self-Attention is $O(n^2)$, this is roughly a 5× increase in compute.

Negative prompt and CFG

In the SD 1.x CFG (Section 7), the empty string $\varnothing$ was used for the unconditional prediction $\bepsilon_\theta(\bz_t, t, \varnothing)$. From SD 2.x onward, a negative prompt $c_{\text{neg}}$ can be specified instead of the empty string:

Geometrically, the CFG guidance vector is the difference between the conditional and unconditional scores, and the negative prompt moves this "reference point" from the empty string to an arbitrary text. The larger the guidance strength $w$, the greater the displacement in the direction from $c_{\text{neg}}$ to $c$.

10. The SDXL Architecture

SDXL (Podell et al., 2023) is an architecture substantially expanded from SD 1.x/2.x.

Dual text encoders

SDXL concatenates the outputs of two text encoders:

In addition, the pooled output of OpenCLIP (1280 dimensions) is added to the time embedding to inject global text information into the model.

Micro-conditioning (size and crop conditioning)

The original size and crop coordinates of the training data are given as additional conditions, and the target size is specified at generation time. They are embedded with Fourier features and added to the time embedding:

This lets the model avoid, at generation time, artifacts such as the "blur" and "unnatural crops" learned from low-resolution training data.

Refiner

The Base model performs the initial large-scale denoising ($t = T \to t_{\text{switch}}$), and the Refiner handles the remaining refinement ($t = t_{\text{switch}} \to 0$), in an ensemble-of-expert-denoisers scheme. $t_{\text{switch}} \approx 200/1000$ is a typical switch point.

11. Diffusion Transformer (DiT / MM-DiT)

SD 3.0 (Esser et al., 2024) abandoned the U-Net entirely and adopted MM-DiT (Multimodal DiT), based on the Diffusion Transformer (DiT; Peebles & Xie, 2023).

Basic structure of DiT

The latent $\bz \in \R^{h \times w \times c}$ is split into $p \times p$ patches and converted by a linear projection into a token sequence $\boldsymbol{T}_{\text{img}} \in \R^{N \times D}$ ($N = hw/p^2$). Positional embeddings are added, and Transformer blocks are repeated.

MM-DiT (the Joint Attention of SD 3.0)

Self-Attention is computed over the sequence formed by concatenating the image and text tokens:

However, the image and text have independent QKV projection matrices (modality-specific feature extraction). Since the Attention computation itself is unified, text and image interact symmetrically.

AdaLN-Zero (adaptive normalization)

In DiT, the timestep $t$ and the text pooled embedding are used to dynamically generate the LayerNorm scale $\gamma$ and shift $\beta$ of each block, as well as the gate $\alpha$ of the residual connection:

Text encoders (triple configuration)

SD 3.x uses three text encoders:

• CLIP ViT-L/14 (768d): basic text understanding
• OpenCLIP ViT-bigG/14 (1280d): high-quality text–image correspondence
• T5-XXL (4096d): understanding of long, complex instructions (language-model based)

The addition of T5 greatly improves fidelity to long natural-language instructions ("a red car parked next to a blue bicycle on a rainy street").

12. Flow Matching and Rectified Flow (SD 3.x / FLUX)

SD 3.x and FLUX leave behind the DDPM framework of "gradually adding/removing noise" and are based on Flow Matching (Lipman et al., 2023) / Rectified Flow (Liu et al., 2023).

Continuous normalizing flow (CNF)

We learn a time-dependent vector field $\boldsymbol{v}_\theta(\bx, t)$ connecting the data distribution $p_0$ and the noise distribution $p_1 = \N(0, I)$. The ODE is:

Data $\bx_0$ corresponds to $t = 0$ and noise $\bx_1$ to $t = 1$. At generation time we solve the ODE from $\bx_1 \sim \N(0, I)$ in the direction $t = 1 \to 0$.

The Flow Matching loss

We define the intermediate state $\bx_t$ by linear interpolation between data $\bx_0$ and noise $\bx_1 \sim \N(0, I)$:

The conditional vector field along this interpolation is:

(a constant-speed straight line connecting $\bx_0$ and $\bx_1$). The Flow Matching loss is:

Rectified Flow (ReFlow)

After one round of Flow Matching training, one resamples pairs $(\hat{\bx}_0, \hat{\bx}_1)$ from the trained model's trajectories and reconnects them with straight lines; repeating this reflow procedure makes the trajectory even closer to a straight line and dramatically improves generation quality at 1–4 steps.

Concrete settings of SD 3.x

The FLUX architecture

Black Forest Labs' FLUX (2024) further develops Rectified Flow + DiT:

• Single/Double stream blocks: Joint Attention of image and text in shallow layers (Double), switching to image-only Self-Attention in deep layers (Single)
• Rotary Position Embedding: applies RoPE to 2D image patches, supporting arbitrary aspect ratios
• Guidance distillation: a distilled model capable of high-quality generation even without CFG (FLUX.1-schnell runs in 1–4 steps)
• CLIP-L + T5-XXL: drops OpenCLIP-bigG from SD 3.x's three encoders, using T5-XXL as the main encoder together with CLIP-L's pooled embedding (the exact configuration can vary by checkpoint and implementation)

Summary

References

• Ho et al., "Denoising Diffusion Probabilistic Models" (DDPM, 2020)
• Song et al., "Denoising Diffusion Implicit Models" (DDIM, 2020)
• Song et al., "Score-Based Generative Modeling through Stochastic Differential Equations" (2021)
• Rombach et al., "High-Resolution Image Synthesis with Latent Diffusion Models" (Stable Diffusion, 2022)
• Ho & Salimans, "Classifier-Free Diffusion Guidance" (2022)
• Nichol & Dhariwal, "Improved Denoising Diffusion Probabilistic Models" (2021)
• Vincent, "A Connection Between Score Matching and Denoising Autoencoders" (2011)
• Salimans & Ho, "Progressive Distillation for Fast Sampling of Diffusion Models" (v-prediction, 2022)
• Podell et al., "SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis" (2023)
• Peebles & Xie, "Scalable Diffusion Models with Transformers" (DiT, 2023)
• Esser et al., "Scaling Rectified Flow Transformers for High-Resolution Image Synthesis" (SD 3.0, 2024)
• Lipman et al., "Flow Matching for Generative Modeling" (2023)
• Liu et al., "Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow" (2023)
• Diffusion model - Wikipedia
• Stochastic differential equation - Wikipedia

Related Articles

• Diffusion Models (Overview)
• Variational Autoencoder (VAE)
• Attention Mechanism
• Transformer
• Generative Adversarial Networks (GAN)

Frequently Asked Questions