Save this video free

Lecture 3: Autoregressive Models (KAIST AIC313/CS378, Fall 2026)

Minhyuk Sung · 1:02:07 · Watch on YouTube

Part of KAIST AIC313/CS378. See all 12 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

Minhyuk Sung explains how autoregressive models make high-dimensional generation tractable by applying the chain rule, then using neural networks to estimate each next-step conditional distribution. The lecture connects exact likelihood and teacher-forced training to historical architectures including 2011 binary-image models, 2015 MADE, causal convolutions, and transformer-based GPT, then outlines image tokenization and multimodal generation.

Key takeaways

Chapters

0:00 Why Long Sequences Make Joint Distributions Intractable
3:25 The Chain Rule Turns a Joint Distribution into Prefix Conditionals
8:35 Choosing an Order Enables Exact Autoregressive Likelihoods
13:50 Neural Networks Parameterize Next-Token Distributions
18:07 Exact Likelihood, Teacher Forcing, and Sequential Sampling
28:00 Early Neural Autoregressive Models for Binary and Continuous Images
36:07 MADE Uses Masks to Predict All Conditionals in Parallel
40:51 Causal Convolutions and Dilation Expand Sequence Context
47:40 Receptive Fields Explain Convolutional Audio Generation
51:29 Two-Dimensional Causal Masks Preserve Image Neighborhoods
55:20 Transformer Attention Uses a Causal Mask for Generation
59:10 GPT, Decoder-Only Models, and Autoregressive Multimodal Generation

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:00 So let's get started with today's topic. So yeah, last time we discussed some kind of the basics of some probability and then uh from this week we're going to start to basically list some kind of the basic kind of the ideas for each of the model. For today's lecture we are going to discuss the order regressive model. Uh so yeah for the today's lecture we're going to discuss like order factoriization learning and sampling and some kind of the basic some kind of the model some deep models from the 201 like 11 or something to the all the way to the some recent models and some transformer based kind of architectures as well you want to see I mean actually the model is kind of the simple in terms of the so let's get back to the kind of the previous kind of discussion in terms

0:49 of learning some kind of the categorical the distribution, right? So, we discussed like the maximum likelihood estimation in terms like how you know when we have some kind of samples, some kind of disc samples, how we can find some kind of the uh compute the maximum likelihood the estimate in terms of giving the best parameters for the categorical the distribution. So, that's kind of the case. But here the question is that uh if we think about some kind of the real case uh for the text generation basically if we're going to make some kind of the GPT then what's the kind of the possibility the cases that we need to consider. So last time I have shown some kind of this very simple kind of the 3x3 table for the two variables like having only the three possibility cases right then we only need to consider the nine possibility

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Minhyuk Sung.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.