CMU Introduction To Deep Learning 11-785, Fall 2026: Lecture 19

Overview

Transformers replace the sequential information flow of RNNs with parallel self-attention, allowing each token to build a contextual representation from other tokens while positional methods such as RoPE encode order. The lecture connects the architecture’s components to training and deployment: next-token pre-training, instruction tuning, LoRA and DPO post-training, Mixture of Experts, and token representations for text, images, and audio.

Key takeaways

Self-attention removes the RNN’s sequential information bottleneck by letting every token directly access other tokens, while enabling parallel computation across positions.
Scaled dot-product attention uses QKᵀ/√dₖ followed by softmax and multiplication by V; the scaling prevents overly sharp distributions and weak gradients.
A causal mask makes next-token predictions safe to compute in parallel during training, while autoregressive inference still generates one token at a time and can reuse cached keys and values.
Pre-training teaches language patterns through next-token prediction, supervised fine-tuning teaches instruction responses, and DPO adjusts preferences using chosen and rejected answers against a reference model.
LoRA and Mixture of Experts solve different scaling problems: LoRA reduces trainable parameters with low-rank updates, while MoE activates only selected feed-forward experts per token.
Transformers can handle images and audio by converting them to vector sequences, as in ViT patch tokens, Whisper spectrogram features, and LLaVA’s projected CLIP image representations.

Chapters

0:00 Lecture Setup and the Transformer Roadmap

The opening minutes include audio, HDMI, and meeting-sharing checks before the lecture begins.
The roadmap covers transformer architecture, attention and position, pre-training and post-training, Mixture of Experts, and multimodal inputs.
Examples include ChatGPT, Gemini, Llama, Whisper, and vision models.

6:35 Contextual Embeddings and the RNN Bottleneck

Static embeddings assign one vector to both meanings of “bank,” while contextual embeddings adapt the representation to surrounding words.
The example “it is tired” versus “it is wide” shows how later words can change the interpretation of an earlier pronoun.
RNNs process tokens sequentially and pass information through hidden states, creating both a parallelization problem and a long-range information bottleneck.
Self-attention lets a token directly gather information from other words, including distant ones.

11:44 The Transformer’s Encoder, Decoder, and Core Components

The original 2017 Transformer has an encoder that processes inputs and a decoder that generates outputs.
Its main components are attention, per-token feed-forward networks, residual connections, normalization, and positional encodings.
Attention provides token-to-token communication; the feed-forward network performs computation independently at each position.
The original architecture stacked six encoder and six decoder layers.

16:09 Self-Attention, Multi-Head Attention, and QKV

In an English-to-French encoder-decoder example, the encoder processes the input once while the decoder generates each output token using causal self-attention and cross-attention.
Self-attention connects tokens within a sequence; multi-head attention runs several learned attention patterns in parallel.
Queries express what a token seeks, keys describe what tokens offer, and values carry the information to combine.
Learned query, key, and value projections let relevance differ from simple embedding similarity.

21:30 Scaled Dot-Product Attention and Why Scaling Matters

Attention computes QKᵀ, divides scores by √dₖ, applies softmax, and multiplies the resulting weights by V.
The lecture’s “The cat sat” example calculates attention weights for “sat” and combines value vectors into a contextual representation.
Scaling controls score growth as key dimension increases, preventing softmax from becoming too sharp and gradients from shrinking.
Attention is parallel across token positions during training, but its score matrix grows quadratically with sequence length.

25:34 Feed-Forward Networks, Residuals, and Causal Masks

A typical feed-forward network expands each token representation from 512 to 2,048 dimensions, applies a nonlinearity, and projects it back to 512.
Residual connections preserve information and gradient paths; layer normalization stabilizes activations across deep stacks.
Modern language models commonly normalize before sublayers, unlike the original Transformer’s post-residual normalization.
Decoder causal masks place negative infinity above the attention diagonal so future tokens receive zero probability, while encoder attention remains unmasked.

32:10 Sinusoidal and Learned Absolute Positional Encodings

Self-attention without position information is permutation-equivariant, so it cannot distinguish “dog bites man” from “man bites dog.”
The original Transformer adds fixed sinusoidal encodings made from sine and cosine waves at different frequencies.
Learned absolute position embeddings assign trainable vectors to positions; BERT and GPT-2 are cited as examples.
A fixed learned position table can make handling sequences longer than its supported range difficult.

36:20 Relative Positions and Rotary Position Embeddings

Relative-position methods encode how far tokens are from one another rather than only their absolute locations.
RoPE rotates pairs of query and key dimensions according to token position, making attention scores sensitive to relative offsets.
The lecture names Llama, Mistral, and Qwen as models using RoPE.
Unlike additive sinusoidal or learned absolute encodings, RoPE incorporates position directly into attention’s query-key interactions.

40:07 Autoregressive Decoding, Sampling, and the KV Cache

Autoregressive generation repeatedly predicts and appends one token based on the prompt and previously generated tokens.
Greedy decoding chooses the highest-probability token; temperature adjusts randomness, while top-k and top-p restrict the sampling set.
Lower temperature suits more deterministic coding responses, while higher temperature can support more varied creative writing.
A KV cache stores prior tokens’ keys and values so generation need not recompute them, trading GPU memory for reduced computation.

45:05 BERT, GPT, T5, and Next-Token Pre-Training

BERT is encoder-only and uses bidirectional context for masked-token prediction; GPT is decoder-only and uses causal next-token prediction.
T5 uses an encoder-decoder architecture to reconstruct masked spans, with cross-attention connecting decoder outputs to encoder representations.
GPT-style pre-training shifts text targets by one token and uses cross-entropy to train next-token predictions at every position.
Raw books, websites, and code supply training targets without manual labels, and causal masking enables parallel predictions across positions.

50:25 Supervised Fine-Tuning and Instruction Tuning

A pre-trained model learns language continuation, but may continue an instruction-like pattern instead of carrying out the requested task.
Supervised fine-tuning trains on instruction-response examples with next-token cross-entropy, typically calculating loss on the assistant response rather than the prompt.
Instruction tuning broadens SFT across tasks such as translation, summarization, explanation, and question answering.
The lecture distinguishes pre-training, which develops language capabilities, from post-training, which teaches the model how to use them in responses.

55:00 LoRA and Direct Preference Optimization

LoRA freezes the base weight matrix and learns a low-rank update through matrices A and B, reducing the number of trainable parameters.
The lecture’s example reduces training from 16 million parameters to about 16,000 with rank-8 LoRA; adapters can target attention projections and feed-forward layers.
LoRA updates can be merged into base weights for inference or kept separate for different tasks, without changing the training objective.
DPO trains on chosen and rejected response pairs against a frozen reference model, directly optimizing preferences without a separate reward-model training stage or reinforcement-learning loop.

1:04:27 Mixture of Experts and Sparse Feed-Forward Routing

A Mixture of Experts layer replaces a dense feed-forward network with multiple expert networks and a router; attention and the other transformer components remain unchanged.
The router scores experts per token and selects a subset, such as the top two, whose outputs are combined using normalized weights.
The lecture cites Mixtral as having about 47 billion total parameters and activating about 13 billion per token.
Sparse routing reduces per-token computation but introduces load-balancing, inter-GPU communication, and memory challenges.

1:13:16 Tokenization and Shared Transformer Inputs

Character tokenization yields long sequences, word tokenization requires a large vocabulary, and subword tokenization balances vocabulary size with sequence length.
Byte Pair Encoding repeatedly merges frequent neighboring pieces into larger tokens.
Because language models process tokens rather than individual letters, tokenization can affect tasks such as counting letters.
Text subwords, image patches, and audio features can all become vector sequences processed by the same attention architecture.

1:16:10 Vision Transformers, Audio Features, and Whisper

A Vision Transformer divides a 224×224 image into 16×16 patches, producing 196 patch tokens, and adds position information plus a classification token.
Audio sampled at 16 kHz produces 16,000 samples per second, so systems use representations such as spectrogram frames, learned acoustic features, or discrete audio codes.
Whisper encodes spectrogram features and uses a decoder with cross-attention to generate text, supporting transcription, translation, and timestamp-related tasks.
wav2vec 2.0 uses contrastive prediction and HuBERT predicts discrete cluster labels, illustrating how self-supervised targets shape learned audio representations.

1:20:00 LLaVA and the Multimodal Assistant Pipeline

A common vision-language pipeline uses a vision encoder, a projector into the language model’s embedding space, and a language model that combines image features with text prompts.
LLaVA uses a pre-trained CLIP vision encoder and projector, first training the projector while freezing the vision encoder and language model.
A second LLaVA stage fine-tunes the language model and projector on visual instruction-following examples.
The closing summary emphasizes attention, positional information, pre-training versus post-training, LoRA, sparse experts, and shared vector processing across modalities.

Transcript excerpt

0:08 Middle East and all of you like to managing that infrastructure. Check check check.

1:18 Yeah, two. doesn't want to change. >> So she said, >> "Yeah, Good luck.

2:45 I got it already. >> Check. Check. >> Okay. Let's just >> You have an HDMI on the other side. >> Uh, >> nope. One second. Sorry. Okay. Good. You're sharing another meeting. It

3:51 should be. >> Yeah. >> Actually, let me You mind if I put the mic closer? All right. >> Okay. Thank you. >> Okay. So let's start. So good morning everyone. >> Um today we're going to talk about the

4:40 architecture behind some of the most powerful AI models you've used such as Chad GPT Gemini Llama Whisper and the vision models behind the image search. So they're built on the same fundamental idea which is the transformers. So what makes this architecture so powerful and why has it has it become the foundation of modern AI? So that's what we're going to understand today. So here's our road map five parts and each answers one question. So the first part is why do we need uh transformers? What were the limitations of earlier architectures

5:26 and how do transformers process sequences differently? Part two, how do transformers work? We'll unpack the architecture, attention, positional encoding and how the model generates during the inference. So this would be the biggest uh part of the lecture. Um and then the third part is how do transformers learn and we'll start with pre-training on row text. Then we will explore how post- trainining turns a language model into an assistant through supervised fine-tuning instruction tuning. We will uh talk about Laura and preference learning. Part four we will learn how do transformers scale. will introduce mixture of experts. A technique that

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Carnegie Mellon University Deep Learning.