Skip to content

SDE Formulation of Diffusion Models

This directory contains comprehensive documentation on the Stochastic Differential Equation (SDE) perspective on diffusion models, providing a continuous-time view that unifies DDPM, DDIM, and other variants.

The SDE formulation reveals diffusion models as continuous stochastic processes, enabling flexible sampling, theoretical analysis, and connections to classical stochastic calculus.

Core Documentation Series

This series mirrors the structure of DDPM and flow matching documentation, separating theory, training, and sampling.

Document Description
00_sde_overview.md Overview: High-level introduction, why SDE view matters, key concepts
01_diffusion_sde_view.md Foundations: Detailed SDE formulation, forward/reverse processes
02_sde_training.md Training: How training works — NO SDE solvers needed!
03_sde_sampling.md Sampling: How to generate samples — SDE/ODE solvers used here

Supplementary Documents

Document Description
01a_diffusion_sde_view_QA.md Design principles Q&A
02a_taylor_expansion.md Taylor expansions in diffusion
02b_fokker_plank_eq.md Fokker-Planck equation derivation
02c_ddpm_to_vpsde.md From DDPM to VP-SDE (reverse direction)
03b_ddim_update_coeff.md DDIM coefficients from theory

Quick Navigation

For Beginners

  1. Start with SDE Overview for high-level understanding
  2. Read SDE Foundations for detailed formulation
  3. See DDPM from SDE for discrete-continuous connection
  4. Understand Solving VP-SDE for exact solutions

For Implementation

  1. DDPM Connection — How DDPM emerges from VP-SDE
  2. DDIM Coefficients — Exact formulas for code
  3. Reverse SDE & ODE — Sampling methods

For Theory Deep Dive

  1. Fokker-Planck Equation — Probability evolution PDE
  2. Taylor Expansions — Mathematical foundations
  3. VP-SDE from DDPM — Continuous limit derivation

Key Concepts

The SDE Formulation

Forward SDE (data → noise):

\[ dx = f(x, t)\,dt + g(t)\,dw \]

Reverse SDE (noise → data):

$$

dx = \left[f(x, t) - g(t)^2 \nabla_x \log p_t(x)\right]dt + g(t)\,d\bar{w} $$

Key insight: Only the score function \(\nabla_x \log p_t(x)\) needs to be learned.

Variance-Preserving SDE (VP-SDE)

The most common formulation, corresponding to DDPM:

\[ dx = -\frac{1}{2}\beta(t) x\,dt + \sqrt{\beta(t)}\,dw \]

Properties:

  • Preserves variance over time
  • Discretizes to DDPM forward process
  • Closed-form marginals: \(q(x_t | x_0) = \mathcal{N}(\sqrt{\bar{\alpha}_t} x_0, (1-\bar{\alpha}_t)I)\)

Probability Flow ODE

Deterministic alternative to reverse SDE with same marginals:

\[ dx = \left[f(x, t) - \frac{1}{2}g(t)^2 \nabla_x \log p_t(x)\right]dt \]

Key property: DDIM is the discretization of this ODE.

Comparison: Discrete vs Continuous

Aspect DDPM (Discrete) SDE (Continuous)
Time Steps \(t = 0, 1, \ldots, T\) Continuous \(t \in [0, T]\)
Forward Markov chain Stochastic process
Notation \(\alpha_t\), \(\bar{\alpha}_t\) (products) \(\beta(t)\), integrals
Sampling Fixed schedule Flexible steps
Theory Discrete probability Stochastic calculus
Flexibility Limited High

Key connection: DDPM is the Euler-Maruyama discretization of VP-SDE.

Learning Path

Conceptual Understanding

  1. SDE Overview — Why SDE view matters
  2. Unified framework for diffusion variants
  3. Flexible sampling strategies

  4. Theoretical foundations

  5. SDE Foundations — Detailed formulation

  6. Forward and reverse SDEs
  7. Score functions
  8. Brownian motion

Practical Connection

  1. DDPM from SDE — Discretization
  2. Euler-Maruyama method
  3. Why DDPM predicts noise

  4. Forward and reverse processes

  5. Solving VP-SDE — Exact solutions

  6. Closed-form marginals
  7. Connection to \(\bar{\alpha}_t\)
  8. Products → integrals

Advanced Topics

  1. Reverse SDE & Probability Flow ODE — Sampling
  2. Stochastic vs deterministic
  3. DDPM vs DDIM

  4. The \(\eta\) parameter

  5. DDIM Coefficients — Theory to code

  6. Exact update formulas
  7. Fast sampling

  8. Implementation details

  9. Fokker-Planck Equation — Advanced theory

  10. Probability density evolution
  11. PDE perspective
  12. Weak vs strong solutions

Interactive Resources

For hands-on learning with code:

  • Location: notebooks/diffusion/02_sde_formulation/
  • Contents:
  • README.md — Comprehensive theory document
  • 02_sde_formulation.ipynb — Interactive notebook with visualizations
  • sde_QA.md — Common questions answered

8 Focused Supplements: 1. Forward SDE Design Choices (VP/VE/sub-VP) 2. Brownian Motion Dimensionality 3. Equivalent Parameterizations (score ↔ noise ↔ \(x_0\)) 4. Training Loss and Denoising 5. Reverse SDE and Probability Flow ODE 6. Fokker-Planck and Effective Drift 7. Fokker-Planck Equation (detailed derivation) 8. Dimensional Analysis

What's Here

This docs/SDE/ directory contains tutorial-style mathematical derivations that complement the interactive notebooks. These documents provide:

  • Detailed line-by-line derivations
  • Mathematical foundations and proofs
  • Historical context and development
  • Advanced theoretical connections

Documents in This Directory

Core SDE Theory

  1. 01_diffusion_sde_view.md — SDE Formulation Overview
  2. Forward and reverse SDEs
  3. Score functions and their role
  4. Brownian motion fundamentals

  5. Connection to diffusion models

  6. 01a_diffusion_sde_view_QA.md — Design Principles Q&A

  7. Why specific drift functions?
  8. Score vs. noise prediction
  9. High-dimensional intuition
  10. Practical design choices

DDPM ↔ SDE Connections

  1. 02_sde_and_ddpm.md — Deriving DDPM from VP-SDE
  2. Euler–Maruyama discretization
  3. Variance-preserving structure
  4. Why DDPM predicts noise

  5. Forward and reverse processes

  6. 02c_ddpm_to_vpsde.md — From DDPM to VP-SDE (Continuous Limit)

  7. Moment matching approach
  8. Taylor expansion of discrete steps
  9. Recovering the continuous SDE
  10. Identity check derivation

Mathematical Foundations

  1. 02a_taylor_expansion.md — Taylor Expansions in Diffusion
  2. Role in Euler–Maruyama
  3. Square root approximation in DDPM
  4. Fokker–Planck equation derivation

  5. Continuous vs. discrete connections

  6. 02b_fokker_plank_eq.md — Fokker–Planck Equation Derivation ⭐

  7. Line-by-line derivation from SDEs
  8. Test function approach
  9. Integration by parts
  10. Weak vs. strong solutions
  11. Examples and intuition

Solving and Sampling

  1. 03_solving_vpsde.md — Solving the VP-SDE ⭐ NEW
  2. Exact solution via integrating factor
  3. Closed-form marginal distribution
  4. Connection: \(\bar{\alpha}(t) = \exp(-\int_0^t \beta(s)\,ds)\)

  5. Discrete products → continuous integrals

  6. 03a_reverse_time_sde_and_proba_flow_ode.md — Reverse SDE and Probability Flow ODE ⭐ NEW

  7. Reverse-time SDE (Anderson, 1982)
  8. Probability flow ODE (Song et al., 2021)
  9. DDPM as SDE discretization
  10. DDIM as ODE discretization

  11. The \(\eta\) parameter

  12. 03b_ddim_update_coeff.md — DDIM Update Coefficients ⭐ NEW

  13. Exact DDIM update formula derivation
  14. From \(\bar{\alpha}(t)\) to alphas_cumprod array
  15. Why \(\sqrt{\bar{\alpha}_t}\) coefficients appear
  16. Fast sampling (skipping steps)
  17. Complete theory-to-code connection

Current status: These documents provide rigorous mathematical foundations that complement the code-focused notebooks.

Notebooks (Interactive + Code)

  • SDE Tutorial (Theory): notebooks/diffusion/02_sde_formulation/README.md
  • SDE Tutorial (Code): notebooks/diffusion/02_sde_formulation/02_sde_formulation.ipynb
  • DDPM Basics: notebooks/diffusion/01_ddpm/01_ddpm_basics.ipynb

Docs (Mathematical Derivations)

Learning Path

For Beginners

  1. Start with SDE View Overview for conceptual understanding
  2. Read DDPM from VP-SDE to see discrete-continuous connection
  3. Understand Solving VP-SDE for the forward solution
  4. See DDIM Update Coefficients for exact code formulas
  5. Work through the interactive notebook: notebooks/diffusion/02_sde_formulation/02_sde_formulation.ipynb

For Deep Dive

  1. Study Taylor Expansions for mathematical foundations
  2. Work through Fokker–Planck derivation for PDE theory
  3. Complete the identity check with VP-SDE from DDPM
  4. Master Solving VP-SDE for closed-form solutions
  5. Understand sampling with Reverse SDE & Probability Flow ODE
  6. Connect theory to code with DDIM Update Coefficients
  7. Explore Design Principles Q&A for practical insights

Topics Covered

  1. Brownian Motion: Properties, scaling, and dimensionality
  2. Forward SDE: Data corruption as a stochastic process
  3. Score Functions: Gradient of log density and its role
  4. Reverse SDE: Time-reversal and sampling
  5. VP-SDE: Variance-preserving formulation (DDPM connection)
  6. Fokker–Planck Equation: Probability density evolution
  7. Discretization: Euler–Maruyama and variance preservation
  8. Moment Matching: Connecting discrete and continuous views

Archive

Note: Original draft files have been moved to dev/diffusion/sde/ (private development area).

These drafts have been superseded by comprehensive tutorials in notebooks/diffusion/02_sde_formulation/supplements/: - Supplement 07: Fokker-Planck Equation (replaces div_and_laplace content) - Supplement 08: Dimensional Analysis (replaces unit_analysis content) - Main sde_QA.md: Canonical Q&A version