Lecture 3: Autoregressive Models (KAIST AIC313/CS378, Fall 2026)
Overview
Minhyuk Sung explains how autoregressive models make high-dimensional generation tractable by applying the chain rule, then using neural networks to estimate each next-step conditional distribution. The lecture connects exact likelihood and teacher-forced training to historical architectures including 2011 binary-image models, 2015 MADE, causal convolutions, and transformer-based GPT, then outlines image tokenization and multimodal generation.
Key takeaways
- The chain rule makes a sequence’s joint probability manageable by expressing it as a product of next-step conditionals, avoiding enumeration of spaces such as the 2^100 outcomes of 100 binary variables.
- Autoregressive models provide exact sequence likelihoods by multiplying their conditional probabilities, and train efficiently with cross-entropy on ground-truth prefixes.
- Teacher forcing parallelizes training across prefix positions, but ordinary autoregressive generation remains sequential because each sampled token conditions the next one.
- MADE’s 2015 connection masks enforce valid autoregressive dependencies while predicting conditional parameters for many positions in one network pass.
- Causal convolutional receptive fields grow through dilation: four kernel-size-2 layers at dilation rates 1, 2, 4, and 8 cover 16 input positions.
- Transformer causal attention, decoder-only prompting, and image latent-patch tokenization extend the same prefix-based generation principle to text and multimodal data.
Chapters
- A table for two variables with three values each has only 3 × 3 = 9 cases, but a binary sequence of length 100 has 2^100 possible sequences.
- A language model with a vocabulary of 100,000 tokens and sequences around 1,000 tokens long cannot enumerate every joint outcome.
- The lecture motivates replacing an enormous joint-probability table with a structured factorization learned by a neural network.
- The chain rule expresses a sequence probability as a product of conditionals, such as p(x1) p(x2 | x1) p(x3 | x1, x2).
- Each token is conditioned on its prefix, matching language generation where previously generated tokens provide context for the next token.
- Limiting context to a local window, such as making x4 depend only on x2 and x3, assumes conditional independence from omitted history given the retained tokens.
- The factorization works for any fixed variable order; text commonly uses left-to-right order, while images can use different pixel and RGB-channel traversals.
- For a short sequence such as “I like cats,” multiplying the relevant conditional probabilities gives its exact joint probability.
- The term autoregressive comes from classical time-series models that predict current values from past values, an idea adapted to generative modeling around 1998.
- Conditioning only on the immediately preceding token is often too restrictive: in “I like cats,” “cats” depends on more than “like.”
- A neural network takes the full prefix or another chosen context as input and predicts the parameters of the next conditional distribution.
- For categorical tokens, the network predicts class probabilities; for continuous values, it can predict Gaussian parameters such as the mean and variance.
- Autoregressive models can compute the exact joint likelihood by multiplying their conditional probabilities, unlike models that provide only a likelihood bound.
- Training minimizes negative log likelihood, equivalent to cross-entropy classification on prefix and next-token pairs drawn from the training sequences.
- Teacher forcing supplies ground-truth prefixes and allows training positions to run in parallel, while generation samples tokens sequentially and can become an inference bottleneck.
- Early deep-learning work around 2011 modeled binary images, including MNIST-style data, by predicting each pixel’s probability of being 1 with a sigmoid output.
- Pixel order matters because different traversals create different conditional dependencies and can be easier or harder for a neural network to learn.
- For continuous image values, a model can predict a mixture of Gaussians, using softmax-normalized mixture weights and positive variances.
- A sequential network could predict each conditional distribution one at a time, but masked connections let one feed-forward pass produce predictions for multiple positions.
- The 2015 Masked Autoencoder for Distribution Estimation, or MADE, masks network connections so each output depends only on permitted earlier inputs.
- The masks enforce autoregressive dependencies while allowing training predictions for all sequence positions to be computed together.
- Causal convolutions predict a token from earlier positions only, often using a limited local window rather than the full sequence history.
- Kernel size controls how many positions a convolution reads, while dilation spaces those positions apart to reach farther into the past.
- Stacking layers with kernel size 2 and dilation rates such as 1, 2, 4, and 8 expands the receptive field without a dense connection to every earlier token.
- With four kernel-size-2 layers and dilation rates 1, 2, 4, and 8, the lecture traces a receptive field spanning 16 input positions.
- Drawing the dependency paths for two layers shows that dilation rates 1 and 1 reach three prior positions, while rates 1 and 2 reach four.
- Causal convolutional models were applied to raw audio generation and could also condition speech output on linguistic features or text.
- Flattening an image into a 1D sequence can make the recent sequence history spatially awkward when predicting a pixel near a row boundary.
- A 2D causal convolution uses a mask that includes only already generated neighbors and excludes pixels that come later in the chosen scan order.
- Implementing the mask carefully preserves the autoregressive constraint while using meaningful local structure in the image’s 2D layout.
- The 2017 Transformer replaced convolutional context modeling with attention while retaining autoregressive ordering for next-token generation.
- A causal attention mask prevents each position from attending to future tokens that have not yet been generated.
- The mask implements the same core constraint as MADE and causal convolutions: predictions may use the prefix, but not future sequence values.
- Encoder-decoder systems can encode user-provided conditions separately, while decoder-only models such as GPT treat the prompt as a prefix and continue generating from it.
- GPT means Generative Pre-trained Transformer; its original framing involved large-scale text pretraining followed by fine-tuning for tasks such as classification and multiple-choice question answering.
- Image-generation systems can tokenize images into latent patches and model those patches as a sequence, making a shared transformer approach possible across text, images, audio, and video.
Transcript excerpt
0:00 So let's get started with today's topic. So yeah, last time we discussed some kind of the basics of some probability and then uh from this week we're going to start to basically list some kind of the basic kind of the ideas for each of the model. For today's lecture we are going to discuss the order regressive model. Uh so yeah for the today's lecture we're going to discuss like order factoriization learning and sampling and some kind of the basic some kind of the model some deep models from the 201 like 11 or something to the all the way to the some recent models and some transformer based kind of architectures as well you want to see I mean actually the model is kind of the simple in terms of the so let's get back to the kind of the previous kind of discussion in terms
0:49 of learning some kind of the categorical the distribution, right? So, we discussed like the maximum likelihood estimation in terms like how you know when we have some kind of samples, some kind of disc samples, how we can find some kind of the uh compute the maximum likelihood the estimate in terms of giving the best parameters for the categorical the distribution. So, that's kind of the case. But here the question is that uh if we think about some kind of the real case uh for the text generation basically if we're going to make some kind of the GPT then what's the kind of the possibility the cases that we need to consider. So last time I have shown some kind of this very simple kind of the 3x3 table for the two variables like having only the three possibility cases right then we only need to consider the nine possibility
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Minhyuk Sung.