Lecture 10: Energy-Based Models 2 (KAIST CS479, Fall 2026)
Overview
Minhyuk Sung connects energy-based model sampling to stochastic differential equations (SDEs), using Brownian motion and the Fokker-Planck equation to explain how drift and noise jointly evolve probability densities. He shows that Langevin dynamics makes the target energy-based distribution stationary, distinguishes that property from convergence toward it, and relates continuous-time SDEs to practical discrete sampling updates.
Key takeaways
- Brownian motion has Gaussian increments with variance equal to the elapsed time, independent future increments, and continuous paths with probability one.
- An SDE for sample motion induces a Fokker-Planck equation for density evolution; in one dimension, constant noise scale G gives ∂tq = −∂x(aq) + ½∂²x(G²q).
- The Fokker-Planck drift term represents probability flow into and out of an interval, while its diffusion term arises from the variance of Gaussian noise.
- Langevin dynamics is designed so the energy-based target density proportional to exp(−E(x))/Z is stationary, allowing samples to be generated without calculating the partition function.
- Stationarity says the target remains unchanged once reached; it is distinct from convergence, which concerns whether sampling from an arbitrary initial distribution reaches the target.
- Small-step discretization converts a continuous SDE into repeated drift-and-noise updates that can be implemented in code such as PyTorch.
Chapters
- An energy function assigns each data point a scalar, with lower energy corresponding to higher probability density.
- Maximum-likelihood training involves an expectation under the learned distribution, which requires sampling from the current model.
- Langevin dynamics combines gradient-based movement with noise, allowing sampling without explicitly computing the partition function.
- The lecture pauses before score matching to explain why sequential noisy sampling can reach a target distribution.
- Brownian motion and SDEs provide mathematical foundations for diffusion models, which the course will cover after the midterm.
- The treatment is intentionally high-level because a full account of stochastic-process theory would require much more than one lecture.
- A one-dimensional Brownian motion starts at zero and describes a random trajectory over time.
- For times s and t, the increment Wt − Ws is Gaussian with variance t − s.
- Future increments are independent of the process history up to time s, and trajectories are continuous with probability one.
- A sequential update combines a drift term, which moves the state in a direction, with a noise term scaled by G.
- As the time step Δt approaches zero, the update becomes an SDE of the form dXt = a(Xt)dt + GdWt.
- The drift a can depend on the current state, while the Brownian increment supplies Gaussian randomness.
- The stochastic-calculus framework is associated with Japanese mathematician Kiyoshi Itô.
- The central question is how the density of Xt evolves when samples follow a specified SDE.
- For generative modeling, the goal is to start from a convenient distribution, such as a standard Gaussian, and use repeated updates to obtain target-distribution samples.
- The Fokker-Planck equation links the SDE for individual samples to a differential equation for their probability density.
- With noise removed and constant velocity set to one, a distribution shifts to the right over time.
- For an interval from L to R, its probability changes according to probability flowing in at one boundary and flowing out at the other.
- Approximating boundary flow as probability density times velocity times Δt gives the one-dimensional drift term, −∂x(aq).
- The derivation uses the fundamental theorem of calculus to turn boundary-current differences into a density derivative.
- With drift removed, Brownian noise spreads an initially concentrated distribution across space.
- A zero-mean Gaussian displacement contributes no first-order change because its mean is zero.
- A second-order expansion retains the variance contribution and yields the diffusion term, one half of the second spatial derivative of G²q.
- Combining drift and noise gives the one-dimensional Fokker-Planck equation: ∂tq = −∂x(aq) + ½∂²x(G²q).
- In multiple dimensions, the drift contribution is expressed using the divergence of the drift multiplied by the density.
- The diffusion contribution uses the Laplacian of the density when the noise scale is constant.
- The key relationship remains: an SDE specifies sample updates, while the Fokker-Planck equation specifies how the density changes.
- Langevin dynamics selects drift and noise terms so that the energy-based target density, proportional to exp(−E(x))/Z, is stationary.
- The discrete Langevin update can be rewritten as an SDE, allowing its density evolution to be checked with the Fokker-Planck equation.
- Substituting the target density into that equation makes its time derivative zero, verifying stationarity for the chosen dynamics.
- Stationarity means that if the sampling density reaches the target distribution, further dynamics leave it unchanged.
- Stationarity alone does not prove that an arbitrary starting distribution will approach the target.
- Convergence is a separate property; the lecture states that Langevin dynamics provides it but leaves its proof and references for further study.
- A continuous-time SDE can be approximated by repeatedly updating the current state over small intervals Δt.
- Each discrete step applies the drift over Δt and adds a correspondingly scaled Brownian-noise increment.
- This discretization turns the mathematical process into sequential updates implementable in PyTorch or similar code.
- Drift alone can concentrate samples at a high-density peak, while noise alone spreads them toward a nearly flat density.
- Combining drift and noise balances concentration and spreading to maintain a structured target distribution.
- Minhyuk Sung says he will share a visualization that lets students vary the energy function and inspect the resulting density evolution.
Transcript excerpt
0:01 So if we briefly recap like what we discussed last time uh the idea basically the energy based model that we discussed last time was basically how we are going to basically model basically all the uh some kind of probability distribution using some kind of the energy function right so the idea of the GAN was basically classifying whether the of the data point is kind of real or the fake basically assign some kind of the binary value for that here the idea A was that we are not just assigning some kind of the binary kind of the classification uh the the output but actually assigning some kind of the one scalar the value which is the function here in a way that converting this into the probability density like this and this is kind of the way like how we define our some kind of the model in
0:49 terms like having those kind of form right so basically what we can see here is that as we have some kind of the lower the energy for the the data point X here that becomes like having some higher the the probability density right so in this way like what we discussed last time was basically to uh like learn those kind of the energy function uh using the maximum likelihood estimation like this way and how we can also sample based on this kind of the energy function right so those are basically two key ideas that we discussed last time so these are the case that the energy function can be given uh with some kind of the whatever some specificity application It can be the case that we are learning from some kind of the training data, right? And if we just try to learn those
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Minhyuk Sung.