Lecture 4: Variational Autoencoders 1 (KAIST CS479, Fall 2026)
Overview
Minhyuk Sung introduces variational autoencoders by contrasting autoregressive models with latent-variable generative models, where a prior and decoder generate data through an unobserved code. Because the evidence requires an intractable integral over latent variables, the lecture derives the evidence lower bound (ELBO) using an encoder to approximate the posterior, then interprets its reconstruction and KL-divergence terms and explains their trade-off.
Key takeaways
- A VAE's evidence p(x) requires integrating the decoder likelihood over latent codes, an operation that becomes intractable for neural-network decoders and impractical to estimate with large numbers of samples for every training example.
- The encoder distribution qφ(z|x) enables the ELBO derivation: Jensen's inequality turns log evidence into a tractable lower bound that can be optimized over both encoder and decoder parameters.
- The ELBO's reconstruction term encourages codes to preserve input-specific information, while KL(qφ(z|x) || p(z)) encourages codes to follow the prior; over-regularization can make distinct inputs map to similar latent distributions.
- The ELBO gap is exactly the KL divergence between the approximate posterior qφ(z|x) and the true posterior pθ(z|x), so the bound is tight when the encoder recovers the true posterior.
- Conditional independence in a decoder does not imply marginal independence: pixels can remain correlated after a shared latent variable z is marginalized out.
Chapters
- Likelihood treats a probability distribution as a function of a variable of interest, such as latent variable z or model parameters θ.
- For fixed model parameters, p(x) can be called evidence or marginal likelihood because it sums or integrates over z.
- Minhyuk Sung emphasizes identifying the variable and conditioning context whenever the term likelihood appears.
- The previous lecture's autoregressive model factors a joint distribution into ordered conditional distributions.
- Masked CNNs and Transformers can evaluate conditionals in parallel, but generate tokens sequentially.
- The lecture shifts from modeling observed-variable sequences to deep latent-variable models.
- KL divergence compares a target distribution P with an approximation Q and measures the expected penalty from using Q.
- Expected negative log likelihood is minimized when the learned distribution matches the true data distribution, assuming sufficient samples.
- KL divergence is zero for identical distributions but is generally asymmetric when P and Q are exchanged.
- Students are asked to calculate KL divergence between two Gaussian distributions and submit the quiz through KMS.
- A separate self-study question asks why KL divergence cannot be negative.
- The lecture connects nonnegativity to Jensen's inequality, introduced after the Gaussian exercise.
- For a convex function, evaluating an interpolated input gives a value no greater than interpolating the function values.
- Jensen's inequality extends this property to expectations: f(E[x]) ≤ E[f(x)] for convex f.
- The logarithm is concave, so its Jensen inequality reverses direction, a key step in deriving a lower bound.
- A latent-variable model first samples code z from a prior, then generates observed data x conditioned on z.
- The approach addresses complex distributions, such as high-dimensional image data, that are poorly represented by a single simple Gaussian.
- A neural-network decoder maps z to parameters of p(x|z), rather than necessarily producing x as a deterministic output.
- A common prior is the standard Gaussian with zero mean and identity covariance, which can be sampled using tools such as PyTorch or NumPy.
- For binary images, the decoder can predict a Bernoulli probability for each pixel and factorize p(x|z) across pixels.
- The decoder's neural network outputs the conditional distribution parameters, including pixel probabilities.
- Factorizing the binary-image decoder assumes pixels are independent given z, not independent after z is marginalized out.
- The lecture illustrates this with two binary pixels whose probability of being 1 is 0.1 for z=0 and 0.9 for z=1.
- A shared latent code can induce correlations between pixels even when the decoder assumes conditional independence.
- Computing p(x) requires integrating p(x|z)p(z) over latent codes z, which is generally intractable when a neural network defines the decoder.
- Monte Carlo estimation approximates the integral by sampling many z values and averaging their contributions.
- Many samples may be needed per training example because most prior samples can produce unrealistic data with very low likelihood.
- The encoder models qφ(z|x), mapping observed data x to a distribution over latent codes rather than to one deterministic code.
- The decoder models pθ(x|z); together the encoder and decoder parameterize inference and generation in opposite directions.
- The true posterior need not be Gaussian even if the prior and likelihood are Gaussian, so the lecture uses a tractable Gaussian approximation, often with diagonal covariance.
- The encoder distribution qφ(z|x) is multiplied and divided into the evidence expression to rewrite it as an expectation.
- Applying Jensen's inequality to the concave log function gives a lower bound on log p(x), the evidence lower bound or ELBO.
- The ELBO can be optimized with respect to decoder parameters θ and encoder parameters φ, avoiding direct computation of the evidence integral.
- The ELBO decomposes into expected reconstruction log likelihood, E_q[log pθ(x|z)], minus KL(qφ(z|x) || p(z)).
- The reconstruction term rewards latent codes that let the decoder assign high likelihood to the original x.
- The KL term regularizes each encoded posterior toward the prior, commonly a standard Gaussian; too much pressure can make different inputs use indistinguishable codes.
- The gap between log evidence and the ELBO is KL divergence from the approximate encoder posterior to the exact posterior, and becomes zero when the approximation matches it exactly.
Transcript excerpt
0:00 it um uh actually before we get started some kind of a new topic actually one things that I wanted to quickly remind you is that uh did you have this kind of question like why we are using uh this term of the likelihood like all different ways uh because if we recall like what we discussed in the lecture two actually I mentioned this term of the likelihood twice in the two different kind of the parts of the slice Yes. So when you talked about the base rule like talking about the prior likelihood and also the evidence and also the posterior I I mentioned the term of the likelihood right uh that's the the case that we having this kind of the conditional distribution uh having the probability x conditioned by z right
0:49 and when you also talk about the maximum likelihood estimation uh for this the case that we are parameterizing uh the probability the density or the mass with the parameters the theta here I also mentioned this as kind of the likelihood uh did you see didn't you see this is kind like weird something or why we use like the same term like in different kind of context was it too clear to everyone or no one had that question I I expect that someone will going to basically raise up this question. No one will guess the this question. No actually we are using the term of the
1:39 likelihood in the same manner for both of the cases. So basically the meaning of the likelihood is basically we are seeing uh this kind of probability the density or the mass function as the function of the something. So basically when we fix the observation the x here uh this is now just becoming
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Minhyuk Sung.