11/18/26: Lect-12
Overview
Joseph Zambreno derives transformer compute costs as roughly 24ND² + 4N²D per layer and explains why both attention computation and its memory demands grow rapidly with context length. He connects those scaling limits to GPT-2, GPT-3, and Llama 3 design choices, including RoPE, RMSNorm, SwiGLU, KV caching, and grouped-query attention.
Key takeaways
- A transformer block’s compute is approximately 24ND² + 4N²D FLOPs: linear projections and feed-forward layers dominate at shorter contexts, while quadratic attention costs become more important as N grows.
- The feed-forward network contributes roughly two-thirds of transformer-block parameters, despite attention receiving much of the architectural attention.
- A 128K-token context creates billions of causal attention entries, making attention memory a major scaling constraint even when only the lower triangle is used.
- KV caching stores previously computed keys and values during autoregressive generation, avoiding repeated work on prior context at the cost of additional memory.
- Grouped-query attention reduces KV cache size by sharing key/value heads among query groups, trading storage efficiency for less query-to-key/value connectivity.
- Llama 3 preserves the decoder-only transformer recipe while adopting RMSNorm, SwiGLU, and RoPE to alter normalization, feed-forward gating, and positional encoding.
Chapters
- N denotes the number of tokens in the context, while D is the embedding dimension used to represent each token.
- A transformer block combines four attention projections, an all-to-all attention calculation, and a two-stage feed-forward network.
- The lecture shifts from parameter counts to the computation and memory required to use scaled models.
- A FLOP is a floating-point operation; in this course, a fused multiply-accumulate, a × b + c, counts as one FLOP.
- Matrix multiplication is cubic in the dimensions of square matrices, corresponding to three nested loops in a basic C implementation.
- Transformer computations are predominantly matrix multiplications, so their costs can be estimated from matrix dimensions.
- The four D × D attention projections cost 4ND² FLOPs, while the two feed-forward layers contribute 8ND² parameters and corresponding multiply-accumulate work.
- Including multiply-accumulate accounting, the linear layers total approximately 24ND² FLOPs per block.
- Attention’s query-key and attention-value operations cost approximately 4N²D FLOPs per block, producing total compute of 24ND² + 4N²D.
- The feed-forward network has about twice as many parameters as attention, so it accounts for roughly two-thirds of block parameters.
- Attention’s share of computation grows as N increases relative to D; historical model scaling often increases both dimensions.
- Across L layers, multiply per-block costs by L; the lecture estimates about 2 teraflops to generate one token with Llama 3 405B.
- An NVIDIA H100’s theoretical peak of roughly 2 petaflops per second implies around 1,000 tokens per second in an idealized estimate; training is often approximated at three times forward-pass compute.
- With D fixed, attention computation grows quadratically with context length N, matching the N² term in the compute equation.
- A causal attention matrix has roughly N²/2 usable entries, but a 128K-token context still implies billions of entries per layer.
- At two bytes per FP16 value, storing those attention scores can require many gigabytes, illustrating the trade-off between recomputing values and retaining them in memory.
- GPT-2 and GPT-3 use the same basic decoder-only transformer design, scaled through model dimensions and depth rather than a wholly different block structure.
- Model parameter estimates include vocabulary-related V × D terms, positional embeddings, and the repeated transformer-block parameters.
- The lecture frames Llama 3 as a well-documented 2024 baseline for studying later transformer changes, while noting that newer architectures may be less openly documented.
- V is vocabulary size; GPT-2 ties its input embedding and output unembedding weights, while GPT-3 uses a separate output table.
- The embedding table was a significant share of GPT-2’s smaller parameter budget but became a rounding error as GPT-3’s transformer blocks grew.
- GPT-3 could add a separate unembedding table with little impact on total model size, unlike the substantial increase it would have caused in GPT-2.
- Llama 3 retains a decoder-only transformer structure while changing selected components, including normalization, activation, positional encoding, and attention.
- RMSNorm scales token representations by their root mean square and uses a learned scale without LayerNorm’s mean subtraction and learned shift.
- SwiGLU adds a gating path to the feed-forward network, using an element-wise product to control which features continue through the network.
- The gated feed-forward design adjusts its dimensions to keep parameter counts near the conventional 8D² budget.
- GPT-2 uses learned positional embeddings added at the input, while rotary positional embeddings, or RoPE, encode position within attention computations.
- RoPE rotates query and key representations so attention depends on the relative distance between token positions, rather than only on absolute sequence positions.
- Position information is applied in each transformer block’s attention, not just once when token embeddings enter the model.
- A decoder-only model generates one token at a time and uses a causal mask so each new token attends only to preceding tokens.
- Previously computed keys and values remain unchanged as generation advances, because the causal rule prevents earlier tokens from depending on future ones.
- A KV cache stores prior keys and values so the model can reuse them instead of recomputing them for every new token; queries for completed steps do not need to be retained.
- Reusing cached context reduces new computation when extending an existing prompt, trading additional storage for less recomputation.
- Grouped-query attention shares key and value heads across groups of query heads, reducing KV cache requirements by approximately the number of sharing groups.
- Compared with standard multi-head attention’s broader query-to-key/value connections, sharing reduces cache storage but also limits connectivity.
- The design trades some potential access to distinct key and value representations for lower memory use; the lecture presents this as a practical efficiency trade-off rather than a free improvement.
Transcript excerpt
0:00 Let's go ahead and get started. Uh where we left off uh on Monday was talking a little bit about the scaling of these models and we did spend a decent amount of time on kind of the parameter bookkeeping of these models which is definitely one part of the equation. When we say uh GPT too small is 125 uh million parameters. The goal of Monday's topic was really just kind of break that down. And I I think the interesting aspect of it is yes, there's a lot of different parts of that transformer-based architecture, but really there's just the two main hyperparameters that kind of dictate everything with regards to the sizing of those models. And yes, the slides use some different variables along the way here. a little sloppy at some points,
0:47 but for the most part, we've been saying N is our context length or length of the number of possible tokens we're processing and d or dcore model is that size of the dimensionality, the embedding size, how many values we're using to represent every single token as they come. And you can refer back to the end of what I called the lecture 11 slides uh for that or Monday's uh video if you weren't here. But really we're kind of able to break it down such that we have this attention mechanism and that is really just uh there's four linear projections that happen and then there's the actual all versus all uh attention matrix multiplication that's happening and then there was the feed forward network and that two-stage feed forward network and both of those in
1:35 terms of the number of parameters can be pretty um concisely described based on both n and E and we did that on Monday. Today I also want to talk about what
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Joseph Zambreno.