Lecture 13: Denoising Diffusion Probabilistic Models 2 (KAIST CS479, Fall 2026)
Watch on YouTube →
Overview
Minhyuk Sung derives the DDPM training objective from maximum likelihood and the evidence lower bound, showing how the known forward Gaussian process and reverse posterior reduce a complex objective to a simple squared-error denoising task. He compares mean, clean-data (x0), and noise (epsilon) prediction, explains why epsilon prediction is practical, and interprets reverse denoising as progressively refining an estimate of the final sample.
Key takeaways
- The DDPM forward marginal q(xt|x0) permits direct sampling at a randomly chosen timestep, avoiding the need to simulate every intermediate noise-addition step during training.
- The ELBO becomes tractable by rewriting forward-transition terms with the posterior q(x_{t-1}|xt,x0); summing over timesteps cancels intermediate terms.
- When the learned reverse Gaussian and target posterior share a variance, minimizing their KL divergence is equivalent to minimizing squared error between their means.
- Practical DDPM training samples x0, t, and xt, then minimizes an L2 prediction loss; schedule-dependent weights are often omitted in the simplified objective.
- Mean, x0, and epsilon prediction are interchangeable parameterizations when their known relationships are used to recover the reverse-transition mean.
- Epsilon prediction is common partly because its target is standard Gaussian noise with a naturally normalized scale, and because it links diffusion training to score matching.
Chapters
- The lecture uses x0 for a data point and xT for the final latent, designed to approach standard Gaussian noise.
- Forward diffusion encodes data by adding noise across Markov steps; generation reverses the sequence from xT toward x0.
- A predefined beta schedule determines Gaussian forward transitions, and the marginal q(xt|x0) allows direct sampling at any timestep.
- Bayes' rule gives q(x_{t-1}|xt,x0), a reverse posterior conditioned on both the noisy state and original data.
- The posterior is Gaussian, with a mean depending on xt and x0 and a variance determined by the forward noise schedule.
- The learned reverse transition aims to match this posterior, but generation does not have access to x0.
- The training goal is maximum likelihood for the data under a model formed by multiplying learned reverse Markov transitions.
- Marginalizing all latent states x1 through xT is intractable, so the lecture applies Jensen's inequality to derive an ELBO.
- The construction follows the same variational lower-bound logic used for VAEs, but includes a full chain of latent variables.
- The reverse model and fixed forward process each factor into conditional distributions across timesteps under the Markov assumption.
- Taking logarithms turns the products into sums, exposing terms involving the terminal prior and reverse transitions.
- A direct KL comparison initially fails because the terms describe different variables, xt and x_{t-1}.
- The key algebraic step rewrites each forward transition using q(x_{t-1}|xt,x0), creating comparisons over the same variable.
- The resulting Gaussian KL terms are tractable, and adjacent log-probability terms cancel when summed over timesteps.
- After cancellation, the objective retains boundary terms and per-step comparisons between the learned reverse transition and the posterior.
- The decomposed ELBO contains a terminal-prior KL, reverse-transition KL terms, and a reconstruction term at the final step.
- The terminal-prior term compares q(xT|x0) with a standard Gaussian and is designed to vanish through the forward schedule.
- The main learnable contribution is matching each reverse transition to the corresponding Gaussian posterior.
- The reverse transition is modeled as a Gaussian whose mean is predicted by a neural network from xt and timestep t.
- Its variance is set to the posterior variance, with a small positive value used at t=1 where the posterior variance is zero.
- When both Gaussians share a variance, their KL divergence reduces to a weighted squared error between their means.
- Ignoring schedule-dependent scaling and constant terms leaves a simple expected L2 denoising loss.
- Each training example samples x0, a random timestep t, and xt directly from q(xt|x0), rather than simulating every forward step.
- The network predicts the target, computes squared error, and updates by backpropagation; Sung notes that the implementation can be only a few lines of PyTorch.
- With only xt available, the optimal reverse mean is the expected posterior mean across all possible x0 values that could have produced xt.
- Generation starts from Gaussian noise and repeatedly samples x_{t-1} using the learned reverse transition until reaching x0.
- Each reverse step uses the current noisy state and timestep, not the original training example.
- Instead of predicting the reverse mean directly, a network can predict x0 from xt and then use the known Gaussian formulas to calculate the reverse mean.
- The x0 output represents an estimate averaged over clean samples compatible with the current noisy state.
- After dropping scaling factors, x0 prediction yields an objective equivalent in form to mean prediction.
- A third parameterization predicts epsilon, the standard Gaussian noise used to construct xt from x0.
- Given xt and a predicted epsilon, the model can recover an x0 estimate and compute the reverse transition mean.
- Epsilon targets are naturally normalized Gaussian samples, which can aid training convergence; pretrained models may instead predict epsilon or x0, so their output interpretation matters during sampling.
- Sung notes that epsilon prediction will connect to score matching and energy-based models in the next lecture.
- Early x0 estimates from highly noisy xt are blurry because many clean samples could plausibly have generated the same state.
- As reverse diffusion reduces uncertainty, the expected x0 estimate becomes clearer and approaches the final generated sample.
- The lecture reframes denoising as repeated refinement: predict the clean outcome, take one reverse step, and update the estimate.
Transcript excerpt
0:01 Okay, so let's continue discussing the diffusion models. So, from the last time we started to discuss the basic ideas of the diffusion models, which is basically have those kind of some progressive the way to generate the final data starting all the way from the the purely Gaussian noise and then gradually denoising that in terms of like generating some final the output like this. And we basically discussed that, you know, uh the diffusion model from the angle of the hierarchical VAE, right? So, in the case like when you uh consider basically, you know, Markovian the hierarchical VAE, uh we could see that the diffusion model, the specific diffusion model that we are seeing here, the DDPM model, uh basically can be seen as kind of the case like having these kind of the two, you know, um sort of
0:49 the properties. The first one is that basically we are having the all the sequence of the latent variables having all the same the dimensionality. And second thing, this is kind of the key part, right? Is that now we are basically having some predefined encoder, basically predefined some kind of forward transition while only learning kind of the reverse part, right? Uh in that way we are kind of like simplifying this kind of framework uh by using some kind of a fixed predefined the forward transition. So, yeah, as we discussed last time, there were some kind of the lots of some kind of the new terms that you'll need to remember uh to, you know, understand all the things, right? So, we basically had those kind of the notations starting from the X0 to the X capital T. So, capital T is basically the number of the the time
1:37 steps, right? In terms of how many kind
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Minhyuk Sung.