11/16/26: Lect-11 (cont.)
Overview
Joseph Zambreno explains how multi-head attention, residual connections, layer normalization, and per-token feed-forward networks fit together in Transformer blocks, then compares encoder-decoder, BERT-style encoder-only, and GPT-2-style decoder-only architectures. He derives the parameter costs, including 4D² for attention and 8D² for the feed-forward network per block, and connects model scaling to training methods such as AdamW and learning-rate warm-up.
Key takeaways
- At fixed model width D, multi-head attention partitions Q, K, and V into H heads without increasing the total projection size; heads let the model learn multiple token relationships in parallel.
- A standard Transformer block uses about 4D² attention parameters and 8D² FFN parameters, so the FFN accounts for roughly two-thirds of the block's dominant weights.
- The dominant parameter count of a vanilla Transformer is approximately 12LD², making width especially costly because model size grows quadratically with D.
- Architecture and masking define the main Transformer families: encoder-only models use unmasked bidirectional attention, decoder-only models use causal attention, and encoder-decoder models combine both with cross-attention.
- GPT-2 combines learned token and position embeddings, causal self-attention, and tied input-output token embeddings; its small configuration has about 125 million parameters.
- AdamW and a learning-rate warm-up followed by decay are practical training tools for stabilizing large Transformers when a fixed learning rate fails.
Chapters
- Mini-project revisions were generally approved; students with a thumbs-up can begin, and Zambreno recommends substantial planning, reading, and prototyping over break.
- Exam 3 covers the final third of the semester, including material through Lecture 12 and related labs, rather than directly testing early-semester topics.
- Exam slots run Wednesday through Friday of week 15; Zambreno recommends choosing Wednesday or Thursday when possible.
- Selected project demos will use most of the class's two-hour final-exam gathering.
- A token such as “it” can relate to earlier tokens through several features, including which noun it refers to and its grammatical role.
- Multi-head attention splits the Q, K, and V projections into H groups so several attention computations can learn different token relationships in parallel.
- At a fixed model width D, splitting into heads does not itself increase the total projection parameter count or computation; it distributes the representation across heads.
- Common head dimensions are around 64 or 128, with model width and head count often scaled together.
- D, also called D_model, is the number of floating-point values used to represent each token after embedding; examples include widths of 512, 768, and 16,384.
- For a token represented by D values, each of W_Q, W_K, and W_V maps from D dimensions to D dimensions using a D × D matrix.
- The context length N counts tokens, while D describes each token's representation; these are distinct dimensions in Transformer computations.
- The number of attention heads H is a separate architecture setting, while each head receives a partition of the projected representation.
- A Transformer block combines self-attention with a per-token feed-forward network, with residual additions and normalization around the sublayers.
- The residual connection adds a block's input x to its output f(x), allowing the model to pass information through when the learned transformation is unhelpful.
- Layer normalization keeps token representations at a manageable scale as they pass through a deep stack of Transformer blocks.
- The original Transformer diagram includes encoder self-attention, decoder self-attention, and cross-attention between decoder queries and encoder outputs.
- Token embeddings map vocabulary entries into D-dimensional vectors, and the original Transformer adds sinusoidal positional encodings so token order is represented.
- The encoder repeatedly processes N tokens through L Transformer blocks; each block returns N token representations of width D.
- The decoder combines causal self-attention with cross-attention to encoder outputs, then projects its final representation to vocabulary-sized logits.
- A softmax distribution supplies the next-token probabilities; generation repeats the process after a token is selected, with temperature or beam search affecting selection.
- Input normalization handles differently scaled features, batch normalization standardizes intermediate activations across batches, and layer normalization operates on each token's feature dimensions.
- The Transformer feed-forward network expands each D-dimensional token representation to 4D, applies a nonlinear activation, and compresses it back to D.
- The attention projections require about 4D² parameters per block, while the two FFN matrices require about 8D², making the FFN roughly two-thirds of these block parameters.
- The dominant parameter and computation costs grow with depth L and approximately with D²; processing all tokens also depends on sequence length N.
- The FFN expansion matrix maps D to 4D and uses 4D² weights; the compression matrix maps 4D back to D and adds another 4D².
- Ignoring biases, the two FFN matrices total 8D² parameters; bias terms add further parameters in architectures that use them.
- GELU, used in Transformer feed-forward networks, is a smooth ReLU-like activation whose common approximation includes the constant 0.044715.
- GELU's added computation is small relative to the matrix operations in a Transformer block.
- The original Transformer is encoder-decoder: the encoder attends bidirectionally without a causal mask, while the decoder uses a lower-triangular causal mask and cross-attention.
- Encoder-only models use unmasked self-attention so each token can use context from both directions, supporting text understanding tasks.
- Decoder-only models apply a causal mask so each token attends only to earlier tokens, supporting next-token text generation.
- Zambreno notes that encoder-only and decoder-only diagrams can look alike; the mask and training objective distinguish their use.
- BERT and RoBERTa are encoder-only models that use bidirectional attention to build representations of existing text rather than generate it one token at a time.
- BERT training masks some input tokens, sometimes replacing them with a mask token or random token, and trains the model to recover the original tokens.
- BERT also used next-sentence prediction, a binary classification objective for deciding whether two sentences follow one another.
- RoBERTa scales and optimizes the BERT-style approach; the lecture notes that next-sentence prediction is no longer commonly used.
- GPT-2 uses decoder-only self-attention with a causal mask, so each position can use preceding tokens but cannot access future tokens.
- Unlike the original Transformer's sinusoidal positional encoding, the GPT-2 diagram uses a learned position-embedding table added to token embeddings.
- GPT-2's Transformer blocks include masked multi-head attention, an FFN, residual connections, and layer normalization.
- Zambreno describes GPT-2 as an influential early model for producing plausible next-token continuations, while noting that access to detailed designs becomes more limited for newer models.
- A token-embedding table for vocabulary size V and model width D contains V × D learned parameters.
- A learned position table for context length N contains N × D parameters; GPT-2's layer norms add learned scale and offset values, typically 2D per layer norm.
- The four attention projections W_Q, W_K, W_V, and W_O contribute 4D² weights and, when present, 4D bias terms.
- The two FFN matrices contribute 8D² weights, with additional biases; GPT-2 ties its output embedding weights to the transpose of its input token-embedding table.
- For a vanilla Transformer, the dominant per-block parameter count is approximately 12D²: 4D² for attention plus 8D² for the FFN.
- Across L blocks, the dominant total scales approximately as 12LD², so increasing model width has a quadratic effect on parameter count.
- GPT-2 small is described as having D = 768 and L = 12, yielding about 125 million parameters; GPT-3 is cited at roughly 175 billion parameters with L = 96 and a much larger width.
- Large parameter counts raise both hardware demands for inference and the much greater computational demands of training.
- Attention projections can be learned through a classification-style objective: produce logits, apply softmax, calculate cross-entropy loss, and backpropagate updates to Q, K, V, and output matrices.
- A small demonstration learns which keys are relevant to each query, improving test accuracy from random performance to 100 percent on its toy task.
- Large Transformer training commonly uses AdamW, combining Adam-style momentum and RMS-scaled updates with decoupled weight decay.
- A learning-rate warm-up starts with small steps, increases the learning rate, then decays it; Zambreno presents scheduling as a practical option when a fixed rate is unstable or ineffective.
Transcript excerpt
0:00 So, in interest of time, let's get started. Um, you may have noticed I haven't provided feedback yet on the mini project uh revisions. Um, I looked at every one of them and they're generally fine. I just wanted to take a little bit more time, probably this evening, to give you feedback. For some of you, the feedback was just thumbs up. So, if you submitted something and you got a thumbs up, just just get started. Um, I guess it goes without saying that you do have to put effort into it, though, right? And it's a different kind of effort than the lab effort. So for the lab effort when you're working on say lab six you you know when you're making progress. I was on section one now I'm on section two. Now I'm on section three. And and for a mini project it's always uh more tricky than that. And I know some of you set uh weekby-eek schedules and I didn't you know focus too carefully on that because
0:48 as soon as you get started usually that schedule kind of gets blown up. But but you want to put some serious time into it. You know for at least a few of those projects. I think if you just took the time that was allocated to your team in lab, I I don't think that would probably be enough. So this this is your final like experience in this particular class, I I think everyone at least the way they wrote, they want to do a good job on those projects. So it's just going to take some time and this could be a good use of your your break time to at least do more planning, more reading, more quick sort of rapid prototyping of
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Joseph Zambreno.