Lecture 14: Score Matching and Diffusion Models (KAIST CS479, Fall 2026)
Watch on YouTube →
Overview
Minhyuk Sung connects DDPM training to energy-based models by showing that predicting Gaussian noise is equivalent, up to scale and sign, to predicting a conditional score; Tweedie's formula also links the score to the expected clean sample. He derives score matching from Fisher divergence using integration by parts, explains why ordinary score matching fails in low-density regions, and shows how Gaussian denoising, multiple noise levels, and annealed Langevin dynamics lead to the Noise Conditional Score Network (NCSN) and clarify its relationship to DDPM.
Key takeaways
- In DDPM, the predicted Gaussian noise is proportional to the negative conditional score, which makes the standard noise-prediction objective a form of score matching.
- Tweedie's formula links the marginal score at a noisy point xₜ to the expected clean sample given xₜ, allowing score, denoised-sample, and noise predictions to be converted into one another.
- Integration by parts turns Fisher divergence into a score-learning objective that can be estimated from data samples without access to the true data score.
- Ordinary score matching underweights low-density regions, even though Langevin sampling may start there; Gaussian perturbation broadens training coverage and supplies useful score targets in those regions.
- NCSN balances broad coverage and score accuracy by training across multiple noise levels, then sampling with annealed Langevin dynamics from large noise to small noise.
- DDPM and NCSN share score-based training and iterative sampling, but DDPM's scaled forward perturbations approach a standard Gaussian prior while NCSN's Gaussian smoothing need not.
Chapters
- The DDPM forward process gradually perturbs data with Gaussian noise until its distribution approaches a standard Gaussian.
- Generation reverses that process, starting from Gaussian noise and repeatedly sampling reverse transitions toward clean data.
- The forward marginal and reverse posterior support an ELBO decomposition, with the terminal prior term vanishing as the final distribution approaches the standard Gaussian.
- A DDPM neural network can predict the reverse-transition mean, the expected clean sample x₀ given xₜ, or the injected noise εₜ.
- Under an L2 training objective, each predictor learns its corresponding conditional expectation given xₜ.
- The forward marginal equation lets expected noise and expected clean data be converted into one another, so the prediction targets are mathematically linked.
- For the Gaussian DDPM forward marginal conditioned on x₀, differentiating its log density gives a conditional score expressible using xₜ, x₀, and εₜ.
- The injected noise is proportional to the negative conditional score, so DDPM noise prediction can be interpreted as score prediction with a scale factor.
- Tweedie's formula relates the marginal score to the expected clean sample given xₜ, connecting score estimates, denoised estimates, and noise estimates.
- An energy-based model defines a target density through an energy function, and its score is the negative gradient of that energy.
- Langevin dynamics updates samples using the score, so sampling requires score information rather than the energy value alone.
- Score matching aims to learn the score directly, avoiding the need to first learn an energy function through maximum likelihood.
- Direct score regression would minimize squared error between a learned score and the data distribution's true score, but data samples do not provide that score directly.
- In one dimension, expanding Fisher divergence isolates a term involving the unknown data score and the learned score.
- Integration by parts transfers the derivative from the data density to the learned score; assuming the density vanishes at infinity yields an objective computable from samples.
- Fisher divergence weights score errors according to the data density, prioritizing accuracy where training samples are concentrated.
- Langevin sampling can begin outside those high-density regions, where an ordinarily trained score may be inaccurate or poorly constrained.
- Bad score estimates at early sampling steps can leave samples without a reliable direction toward the data distribution.
- Denoising score matching adds Gaussian noise to data samples and learns the score of the resulting smoothed distribution rather than only the original data distribution.
- A perturbed sample is generated by selecting a real data point and adding Gaussian noise, making training possible from the original samples.
- The conditional Gaussian score is analytically available, allowing the marginal-score objective to be trained through a conditional-score matching loss.
- At each diffusion time step, DDPM perturbs data with Gaussian noise and trains a network to predict that noise.
- Because noise prediction represents a scaled negative conditional score, DDPM training matches the score of each perturbed distribution.
- The shared training structure connects DDPM's ELBO-based derivation to score matching for energy-based models.
- Large Gaussian smoothing covers broader regions and provides score estimates away from data samples, but its score differs more from the original data score.
- Small smoothing better approximates the original score but provides less coverage in low-density regions.
- Using multiple noise levels lets one score network learn both broad, coarse directions and more accurate local scores.
- The Noise Conditional Score Network trains a single neural network conditioned on the noise level to estimate scores across multiple smoothed distributions.
- Its objective sums score-matching losses over noise levels, with weighting to account for the larger conditional-score magnitudes at smaller noise scales.
- Annealed Langevin dynamics begins with the largest noise level to guide samples across broad regions, then decreases noise to refine them near the data distribution.
- DDPM uses a scaled Gaussian forward process whose terminal distribution approaches a standard Gaussian, providing a natural starting prior for reverse generation.
- NCSN adds Gaussian noise at several scales without the same mean-scaling structure, so its most perturbed distribution need not be standard Gaussian.
- Both methods learn noise-conditioned scores and use iterative sampling; their key distinction is how their forward perturbation distributions are designed.
- The lecture closes by noting that other forward marginals can define diffusion processes if they ultimately connect data to a usable prior distribution.
Transcript excerpt
0:01 So, how did you like the our the discussion for the the diffusion model so far? Is the idea clear? All the things about diffusion models. So, as you can see like you know, we are getting into the slightly more kind of complicate kind of things, right? So, we discussed the basics of the diffusion model, especially the basic model which is called the DDPM and how the diffusion model the idea of the diffusion models are related to the VAE in terms of the like minimizing the elbow function. And today we are also going to see how the diffusion models are kind of related with the energy based models as well in terms of the this core matching. And we are also see the connection between diffusion model and also the flow models
0:49 in the later day lectures. So, yeah, that's kind of you know, things that we are going to discuss after midterms. So, with those kind of the diffusion models and also the flow based model as kind of the center we are going to see how this is kind of like related to the all the models that we have discussed so far before the midterm. So, yeah, this was kind of the the basic idea in terms of that we are gradually basically perturbing our some kind of the data distribution in terms of adding some sort of the Gaussian noise in a way that at the end we are going to get some kind of the standard Gaussian distribution like this and then in the generation that process we are putting into the reverse way, right? If we start from the Gaussian this sample and we gradually basically, you know, sample
1:36 the new kind of the ones in terms of like over the time for those kind of
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Minhyuk Sung.