CMU Introduction To Deep Learning 11-785, Fall 2026: Lecture 18

Overview

The lecture builds from recurrent sequence-to-sequence translation models to Transformers, showing how attention addresses information loss through dynamically weighted context vectors and how self-attention removes recurrence. It connects query-key-value attention, multi-head layers, positional encodings, and masked decoding to the 2017 Transformer, then explains how decoder-only GPT and encoder-only BERT arise from the same architecture.

Key takeaways

A single final encoder state is a poor summary for long inputs; attention instead recomputes a source-weighted context for every output token.
Query-key matching determines which input positions matter, while value vectors carry the content combined into the context.
Self-attention removes recurrence by letting every token build a context-sensitive representation from other tokens, with an N-token sequence requiring an N by N attention-score matrix.
Positional encodings are essential because self-attention alone does not distinguish token order; sinusoidal encodings make relative position gaps available to attention.
Masked self-attention prevents a Transformer decoder from seeing future target tokens, preserving autoregressive generation while enabling parallel training computations.
GPT and BERT use different portions of the Transformer design: GPT is decoder-only for generation, while BERT is encoder-only for input representation learning.

Chapters

0:00 Lecture Roadmap: From Seq2Seq Models to Transformers

The class opens by setting a progression from the simple translation model covered previously to Transformers.
At 4:38, the lecture begins its review of sequence-to-sequence models and teacher forcing.

4:38 Seq2Seq Recap: Encoder, Decoder, and Teacher Forcing

An RNN encoder processes the input sequence into a representation, and an RNN decoder uses it to generate an output sequence.
Teacher forcing feeds the correct previous output into the decoder because sequence probability factors into next-symbol probabilities conditioned on prior symbols.
At inference, outputs feed back autoregressively, beginning with a start-of-sequence token and continuing until an end-of-sequence token.

10:00 Why Encoder-Decoder RNNs Lose Long-Range Information

The decoder's initial input representation can be diluted by subsequent recurrent updates, leaving later outputs with less information about the source.
The encoder's final hidden state must compress the whole input, so early words in long sentences can be forgotten.
Reversing the source sequence was an early workaround that gave initial words stronger influence, but did not remove the bottleneck.
Translation alignments can be nonlocal: in “I ate an apple” to German, “ate” contributes to both “habe” and “gegessen.”

16:00 Attention Context Vectors Replace a Single Encoder Summary

Passing encoder information to every decoder step helps prevent source information from fading during decoding.
A single average of encoder states avoids relying only on the final state, but treats every input position as equally relevant.
Attention instead creates a distinct weighted context vector for each output step, allowing “apple” to focus on its corresponding source word.
Attention weights must be computed dynamically so their focus can change with the decoder's current state.

28:00 Attention Scores, Softmax, and Context Computation

Each raw attention score compares an encoder hidden state with the latest decoder state, which summarizes the output history available at that step.
A dot product works when encoder and decoder states have matching dimensions; a learned matrix projection supports different dimensions.
Softmax converts the input scores into positive weights that sum to one, making the context a convex combination of encoder states.
The decoder recomputes scores, attention weights, and a weighted context at every output step.

37:00 Query, Key, and Value Vectors Separate Matching from Content

The decoder produces a query, while each encoder position produces a key for calculating how relevant that position is.
A separate value vector carries the detailed source information that contributes to the context after attention weights are calculated.
Learned linear transformations generate queries, keys, and values from hidden states; keys determine where to attend, and values determine what information is combined.
The lecture distinguishes matching by broad category from preserving exact content, such as identifying food as relevant while retaining that the word is “apple.”

43:00 Conditional Sequence Probabilities and Beam Search

The model assigns each output token a probability conditioned on the source and all earlier output tokens.
The probability of a complete sequence, such as ABC, is the product of the successive conditional probabilities for A, B, and C.
Beam search keeps the top K partial output sequences and expands them, rather than exploring the full branching tree.
Each beam hypothesis has its own decoder history and attention context, so its next-token probabilities are path-specific.

47:20 Attention Maps Reveal Translation Alignments

Translation attention maps show which English source words receive weight for each generated French word, with lighter regions indicating higher attention.
Examples align “la” with “the” and “accord” with “agreement,” while also tracking reordered words in “la zone économique européenne.”
The learned attention pattern can follow source-target correspondences without manually specifying translation rules.

51:00 Maximum-Likelihood Training and Teacher Forcing

Training uses paired source and target sequences, passing the source through the encoder and the correct target history into the decoder.
Maximum-likelihood learning raises the probability of the correct next token, commonly using a divergence or cross-entropy loss at each step.
Teacher forcing gives the decoder correct prior tokens during training, even though inference must rely on its own generated tokens.
Scheduled sampling can occasionally feed a sampled prediction instead; a Gumbel-softmax technique provides a differentiable approximation to categorical sampling.

58:00 Multi-Head Attention and Visual Attention Examples

Multi-head attention learns separate query, key, and value projections so different heads can capture different input relationships.
The head-specific context vectors are concatenated and passed to the decoder; the lecture notes that heads are not explicitly forced to specialize.
Image-captioning examples show attention focused on a Frisbee when generating “Frisbee” and on a stop sign when generating “stop.”
A giraffe-captioning example directs attention toward the surrounding trees when the caption describes trees.

1:02:00 Self-Attention Lets Each Input Token Build Context

Once a decoder can attend to all encoder positions, the encoder no longer needs recurrence merely to carry information forward.
A token first receives an initial representation, then uses its query to attend to keys across the input and combine their values.
The updated representation for “apple” can incorporate “ate,” helping distinguish the fruit from the Apple computer.
Self-attention updates every position using information from the other positions in the same sequence.

1:07:20 Transformer Blocks: Multi-Head Self-Attention and MLPs

Multi-head self-attention gives each token several attention-based representations, which are concatenated and mixed by an MLP.
The lecture identifies the MLP as a high-parameter component that stores much of a language model's learned information.
For a sequence of length N, self-attention forms an N by N matrix of token-to-token attention scores.
Unlike recurrence, attention can compute token interactions in parallel using matrix operations, avoiding step-by-step hidden-state updates.

1:11:30 Sinusoidal Positional Encodings Add Token Order

Self-attention alone does not encode word order, so permuting tokens can leave its basic attention calculation unchanged.
Positional encodings are added to token representations to make attention sensitive to sequence position and relative distance.
The lecture describes sinusoidal vectors whose inner-product relationships depend on the gap between positions, not just their absolute indices.
The original Transformer uses sine and cosine functions at multiple frequencies; other positional schemes can also be learned or designed.

1:19:20 Masked Self-Attention Preserves Autoregressive Decoding

An encoder can attend to both earlier and later tokens, but an autoregressive decoder must not use future target tokens.
Masked self-attention sets future-position weights to zero and normalizes over only the available past positions.
Transformer decoder blocks combine masked multi-head self-attention with an MLP and can be stacked into multiple layers.
This design replaces recurrence while preserving next-token prediction's causal constraint.

1:22:30 The 2017 Transformer: Parallel Attention Replaces Recurrence

The lecture identifies “Attention Is All You Need,” by Ashish Vaswani and colleagues, as the 2017 paper introducing the Transformer.
The paper's rationale was that attention permits parallel computation across sequence positions, whereas recurrence processes positions sequentially.
The lecture notes that the Transformer achieved strong performance while substantially reducing computation time relative to recurrent approaches.

1:24:00 GPT and BERT as Decoder-Only and Encoder-Only Transformers

When input and output share a vocabulary, a decoder can condition on an initial text prefix and generate the continuation, yielding the GPT-style decoder-only setup.
Task conditioning and larger parameter counts extend the generative model family represented by systems such as ChatGPT and Claude.
When input and output are fundamentally different, such as speech and text, a separate encoder can produce input representations.
An encoder used on its own for representation learning gives the BERT-style encoder-only setup.

1:25:50 Why Explicit Attention Often Beats Recurrent Memory

A student asks whether recurrence could replace sinusoidal position information; the response is that it can, but recurrent models can fade information according to learned parameters.
The lecture contrasts explicit attention to distant context with the more implicit memory mechanism in RNNs and LSTMs.
Recurrent models remain an active research direction, but choosing what distant context to retain can be more challenging than in attention-based models.

Transcript excerpt

2:02 Mhm. >> Hey.

4:35 >> Begin the class. We're going to be talking about continuing on the topic of sequence to sequence models. And today we're going to build up all the way from the simple translation model that we spoke of in the last class until transformers. So Shut your laptops after you're done. Does anybody want to remind me what the last thing we covered was? So okay, anybody else want to remind me what Yeah, Jacob's right. Anybody else

5:22 want to remind me what the last thing we covered was? Anybody? Mhm? What was the topic? Overall? Language models. We were looking at simple translation models. How did we define the simple translation model? Can you speak up, please? But how did How was the model actually constructed? What did it do? Yeah, I'm getting into the little bits, but the overall structure was that we had an encoder which processes processed the entire input and then generated a representation of the entire input. Then

6:10 we had a decoder which sort of grabbed this input with this hidden representation and then unravelled it into the output sequence. And that's how we got a sequence-to-sequence model. A sequence went in and a sequence came out, right? Now >> With teacher forcing. >> With teacher forcing. What did teacher forcing do and how did we explain it? >> We included the output from each step into the next output's model. >> Yes, and how how what was our explanation for this? How did we justify this? >> How do we justify it? >> Uh-huh. >> To keep it aligned. >> That is the mechanics of it. How did we justify it conceptually? >> Conditional probabilities. >> Conditional probabilities. We said this is a language model and when you do this

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Carnegie Mellon University Deep Learning.