Lecture 8: Generative Adversarial Networks (KAIST CS479, Fall 2026)
Overview
Minhyuk Sung compares autoregressive models, normalizing flows, and VAEs before explaining how Generative Adversarial Networks (GANs) learn through competition between a generator and a real-versus-fake discriminator, rather than by directly maximizing likelihood. He derives the binary cross-entropy objective, connects the optimal discriminator to the Jensen-Shannon divergence, and covers practical challenges including vanishing gradients, unstable minimax training, and mode collapse.
Key takeaways
- A GAN can train without directly computing likelihood: its generator learns through a discriminator that distinguishes real data from generated samples.
- With equal real and fake sampling rates, the optimal discriminator is pdata(x) / (pdata(x) + pg(x)); outputs above or below 0.5 indicate local underrepresentation or overrepresentation by the generator.
- Under the assumption of an optimal discriminator, the original GAN minimax objective corresponds to minimizing Jensen-Shannon divergence between data and generated distributions.
- The original minimax generator loss can saturate when the discriminator confidently rejects fake samples; maximizing log D(G(z)) gives the generator a stronger gradient in that situation.
- GANs may generate sharp, realistic samples yet suffer mode collapse, so visual quality of individual outputs does not establish that the model captures the full data diversity.
- GAN training alternates between updating the discriminator and generator while freezing the other network's parameters, and practical stability often depends on additional engineering.
Chapters
- Stacked normalizing-flow layers and variable permutations increase expressiveness, but flows require invertible mappings and tractable Jacobian determinants.
- Autoregressive models can use Gaussian mixtures or discrete distributions such as binary and categorical outputs.
- Autoregressive models and flows can maximize likelihood directly, while VAEs optimize a lower bound on evidence.
- Model choice depends on the data and task; no single approach is best in every case.
- The Jensen-Shannon (JS) divergence compares distributions P and Q through their mixture M = 0.5P + 0.5Q.
- Like KL divergence, JS divergence is nonnegative and reaches zero when P and Q match.
- Unlike KL divergence, JS divergence is symmetric and has a finite upper bound.
- A GAN maps a latent code, typically sampled from a Gaussian distribution, to a data sample using a deterministic generator.
- Unlike a normalizing flow, the generator does not need to be invertible or provide a tractable Jacobian determinant.
- GAN training uses a discriminator objective instead of directly evaluating the data likelihood or its lower bound.
- Minhyuk Sung recalls the shift from small, manually tuned neural networks to deep networks trained with backpropagation.
- AlexNet and convolutional neural networks helped drive ImageNet classification accuracy above 80 percent.
- The gains in recognition performance raised a new question: whether neural networks were robust to small input perturbations.
- Near-identical images can produce different classifications after a small, carefully chosen perturbation.
- The lecture illustrates adversarial examples that make a classifier label varied images as an ostrich.
- Adversarial training uses perturbed examples as additional training data to improve classifier robustness.
- The discriminator classifies inputs as real training data or generated samples.
- The generator transforms latent codes into images designed to fool the discriminator.
- Training both networks creates an adversarial competition: the discriminator improves its judgments while the generator improves its samples.
- The discriminator uses binary cross-entropy to assign high real probabilities to data samples and low real probabilities to generated samples.
- Its maximization objective is E over real x of log D(x), plus E over latent z of log(1 - D(G(z))).
- The generator is trained in the opposing direction, increasing the discriminator's real-class probability for generated samples.
- With the generator fixed and equal real and fake sampling rates, the optimal discriminator is D*(x) = pdata(x) / (pdata(x) + pg(x)).
- If D*(x) exceeds 0.5, the data density is higher than the generator density in that region, indicating underrepresentation.
- If D*(x) is below 0.5, the generator overrepresents that region; if the distributions match, the discriminator outputs 0.5.
- Substituting the optimal discriminator into the GAN objective relates the generator's minimax objective to Jensen-Shannon divergence.
- The derivation explains the objective's distribution-matching effect, but assumes an optimal discriminator.
- Practical training alternates updates to the discriminator and generator rather than solving for a perfect discriminator at each step.
- A discriminator step updates discriminator parameters while stopping gradients through the generator.
- A generator step passes generated samples through the discriminator but updates only generator parameters.
- Minhyuk Sung recommends implementing a GAN to understand the alternating optimization and the engineering needed for stable training.
- When the discriminator easily rejects early generated samples, the original minimax generator loss can provide gradients close to zero.
- The lecture presents a non-saturating alternative: train the generator to maximize log D(G(z)), preserving a useful learning signal when D(G(z)) is small.
- Changing the generator loss can help, but does not eliminate the broader instability of GAN minimax optimization.
- GANs can produce high-resolution, realistic images, as illustrated by StyleGAN2, but training often requires substantial engineering.
- Mode collapse occurs when the generator captures only part of a multimodal distribution; mode hopping can shift samples between modes without covering them all.
- Evaluation should consider both sample realism and diversity across many generated examples.
- The lecture previews diffusion models as a way to pursue high quality and diversity while sacrificing sampling speed.
Transcript excerpt
0:02 Yeah. So basically this quiz shows like multiple things. So basically as we discussed last time with this kind of the gausian the base the one layer of the offline the model can be actually interpreted especially the gaus and the typical a model right so in the sense it sounds like normalizing flow seems kind of like something that's like better than the or model because now we can stack multiple these kind of the flow layers in terms like while basically permuting the order the variables right so in terms of basically permitting the order of the variables. Now we can increase the expressiveness of the D model. Right? So it sounds like the normalizing flow is kind of the better than the original model. But actually that's not exactly the case because this
0:49 is the the case that we only compare the normalizing flow with the Gaussian base the ARD model for the orderless model. We can basically model the conditional distribution not as kind of the single gausian but also with the gausian the mixture and also other types of the distribution. It can be the binary distribution and also the the categorical distribution right. So the good thing of the model is that we can also easily handle like all the discrete data as well. So that's kind of also the difference between the the model and the normalized flow. Right? So we are seeing some kind of the uh some kind of some common things and also kind of differences between these two cases and also when you compare the the normalizing flow with the BA also the normalizing flow might look
1:37 kind of be better in terms of that that's basically the model that allow us to compute the exact likelihood. So like
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Minhyuk Sung.