Lecture 5: Variational Autoencoders 2 (KAIST CS479, Fall 2026)
Overview
Minhyuk Sung explains how variational autoencoders optimize the evidence lower bound (ELBO) using a reconstruction term and a KL-divergence regularizer, with Gaussian assumptions making the KL term analytically tractable. He connects the objective to encoder and decoder design, reconstruction losses, the reparameterization trick, and the tradeoff between faithful reconstruction and useful generation, including posterior collapse and β-VAE weighting.
Key takeaways
- A VAE's ELBO balances expected reconstruction likelihood against KL divergence from each encoder posterior to a standard Gaussian prior.
- For binary observations, decoder negative log-likelihood becomes binary cross-entropy; for continuous observations modeled with a fixed-variance Gaussian, it becomes scaled squared error.
- The reparameterization trick writes z = μ + σ ε with ε sampled from a standard Gaussian, allowing reconstruction gradients to reach encoder parameters.
- Posterior collapse occurs when the KL penalty makes input-conditioned latent distributions nearly identical to the prior, so z carries little information about the input.
- Reducing β emphasizes reconstruction, while increasing β emphasizes prior matching; either extreme can undermine the desired balance between accurate reconstruction and useful generation.
- A conventional autoencoder may reconstruct training examples well but generate poorly from prior samples because its latent codes are not regularized to match that prior.
Chapters
- A standard Gaussian prior and a Gaussian approximate posterior make KL divergence available in closed form.
- The decoder defines a likelihood for observations generated from latent code z, but computing evidence requires an intractable integral over latent variables.
- An encoder approximates the posterior, allowing training to maximize the ELBO instead of directly maximizing the evidence.
- The ELBO combines a reconstruction term with a KL term that regularizes the encoder's approximate posterior toward the prior.
- The reconstruction term trains both encoder and decoder, while the KL term acts directly on the encoder's distribution.
- Generation needs only a prior sample and the decoder; the encoder is used to learn the model during training.
- A binary latent variable example uses prior probabilities of 0.25 for z=0 and 0.75 for z=1, plus conditional likelihoods for a binary observation x.
- Bayes' rule combines the prior and likelihood to compute evidence and the posterior probability of z after observing x=1.
- An observation can make z=0 more probable even when z=1 was more probable under the prior alone.
- The lecture focuses on a standard Gaussian prior and a diagonal Gaussian approximate posterior, regardless of whether observations are discrete or continuous.
- A Gaussian prior and Gaussian decoder likelihood do not guarantee that the true posterior is Gaussian.
- The encoder predicts a mean and log-variance for each input; predicting log-variance ensures the resulting variance is nonnegative.
- A more expressive posterior could approximate the true posterior better, but would require more parameters and make training harder.
- For conditionally independent binary observation dimensions, the decoder predicts probabilities and the negative log-likelihood becomes binary cross-entropy.
- For continuous observations, a Gaussian decoder can predict a mean while keeping its variance fixed.
- With fixed Gaussian variance, the negative log-likelihood reduces to a scaled squared-error reconstruction loss.
- Learning the decoder variance can destabilize training, so the lecture describes fixing it as a practical simplification.
- The reconstruction objective averages the decoder log-likelihood over latent codes sampled from the encoder distribution.
- Multiple z samples can estimate this expectation more accurately, but each additional sample requires another decoder evaluation.
- A common efficient approach takes one latent sample per input while using a minibatch of multiple observations.
- Unlike a deterministic autoencoder, a VAE reconstructs through a sampled latent code rather than a single fixed encoding.
- The KL term pushes the encoder's diagonal Gaussian posterior toward the standard Gaussian prior, whose mean is zero and variance is one.
- The closed-form Gaussian KL lets training regularize the encoder using its predicted mean and variance without estimating the divergence by sampling.
- The Gaussian posterior is chosen primarily for computational simplicity, not because the true posterior is guaranteed to have a Gaussian shape.
- More expressive posterior families could improve approximation if their KL terms can also be computed or estimated effectively.
- Good reconstruction requires each latent distribution to retain information about its input, so the decoder can infer which x produced z.
- The KL penalty pushes all input-conditioned posteriors toward the same standard Gaussian, making latent codes less distinguishable.
- If the KL term dominates, posterior distributions can collapse toward mean zero and variance one, leaving z with little information about x.
- The two ELBO terms therefore compete: reconstruction separates representations while KL regularization encourages overlap.
- Directly sampling z from the encoder distribution interrupts the gradient path from reconstruction loss back to encoder parameters.
- The reparameterization trick samples ε from a standard Gaussian and computes z = μ + σ ε, making z a differentiable function of encoder outputs.
- Holding ε fixed during a gradient update allows backpropagation through μ and σ while preserving the intended sampling distribution.
- Training then combines decoder reconstruction loss with the analytic KL term and updates both networks.
- For each minibatch, the encoder predicts mean and log-variance, Gaussian noise generates z, and the decoder produces a likelihood for the reconstruction loss.
- The encoder is unnecessary at generation time: sample z from the prior and pass it through the decoder to generate x.
- A conventional autoencoder can reconstruct well but has no guarantee that its latent codes match the prior, so prior samples may decode to unrealistic outputs.
- A VAE regularizes its latent space for sampling, though reconstructions may be slightly worse than those of an unregularized autoencoder.
- The standard ELBO uses β=1; lowering β emphasizes reconstruction and approaches ordinary autoencoder behavior as β approaches zero.
- Increasing β strengthens prior matching but can make reconstruction harder and cause posterior collapse, where different inputs receive nearly identical latent distributions.
- A collapsed decoder may produce similar average outputs even when the total objective decreases, so reconstruction and generation quality must be checked separately.
- Choosing β is a practical balance between preserving input information and making prior samples useful for generation.
- When nearby observations have overlapping approximate posteriors, a decoder may produce an average of those observations, contributing to blurry outputs.
- Some overlap lets a generative model produce plausible new examples rather than only retrieving exact training samples; too much overlap can erase distinctions.
- Excessively separating posteriors can improve reconstruction but reduce the model's ability to generate novel combinations.
- The lecture closes by flagging restricted encoder capacity and posterior collapse as additional limitations, then previews normalizing flows as the next model topic.
Transcript excerpt
0:01 Yeah. So yeah, we discussed the K divergence last time and the most important thing in the discussion of the BA would be is that how we can really compute this KD divergence especially for the cases like when the these two distributions are defined as the Gaus and the distribution. Obviously for the other distributions we might want to know how we can compute this. uh but for many other cases uh it may not be you know there would be there would not be some kind of the some closed form uh for the k divergence but the thing is that for the simple the gausian distribution we can actually derive this kind of closed form the formulation for the k divergence so we're going to use this formulation uh for the many parts of like computer training the okay so actually that's kind of thing uh when it
0:48 comes to like using the gaus and the distribution uh many things became kind of the becomes like comfortable in terms of like we can derive those kind of closed form then we can compute all those kind of things but if we start to work on some like more the complexity distribution then there's no way to compute uh those things in a kind of like analytically way so that brings lots of the challenges and that's kind of the one of the many reasons like why we play with the gausian distribution so we talked about that we're going to use the uh standard gausian as the priority distribution And then also we're going to define the approximate the posterior also as the gausian distribution as well. Right? And based on that we're going to use those kind of use this kind of the formulation uh to measure the some the the quantity
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Minhyuk Sung.