Save this video free

11/18/26: Lect-12

Joseph Zambreno · 58:02 · Watch on YouTube

Part of Iowa State EE/CPRE/CYBE 3250. See all 6 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

Joseph Zambreno derives transformer compute costs as roughly 24ND² + 4N²D per layer and explains why both attention computation and its memory demands grow rapidly with context length. He connects those scaling limits to GPT-2, GPT-3, and Llama 3 design choices, including RoPE, RMSNorm, SwiGLU, KV caching, and grouped-query attention.

Key takeaways

Chapters

0:00 Transformer Size Depends on Context Length N and Width D
2:50 FLOPs and the Cost of Matrix Multiplication
5:50 Deriving Transformer Compute: 24ND² + 4N²D
10:25 Compute Shares, H100 Throughput, and Training Cost
16:00 Why Long Contexts Strain Attention Memory
21:20 GPT-2 and GPT-3 Share a Scalable Decoder-Only Recipe
27:00 Vocabulary Tables and Parameter Growth from GPT-2 to GPT-3
31:50 Llama 3 Replaces LayerNorm with RMSNorm and Uses SwiGLU
38:55 RoPE Applies Relative Position Information Inside Attention
47:35 Autoregressive Generation Reuses Prior Keys and Values
54:25 Grouped-Query Attention Reduces KV Cache Storage

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:00 Let's go ahead and get started. Uh where we left off uh on Monday was talking a little bit about the scaling of these models and we did spend a decent amount of time on kind of the parameter bookkeeping of these models which is definitely one part of the equation. When we say uh GPT too small is 125 uh million parameters. The goal of Monday's topic was really just kind of break that down. And I I think the interesting aspect of it is yes, there's a lot of different parts of that transformer-based architecture, but really there's just the two main hyperparameters that kind of dictate everything with regards to the sizing of those models. And yes, the slides use some different variables along the way here. a little sloppy at some points,

0:47 but for the most part, we've been saying N is our context length or length of the number of possible tokens we're processing and d or dcore model is that size of the dimensionality, the embedding size, how many values we're using to represent every single token as they come. And you can refer back to the end of what I called the lecture 11 slides uh for that or Monday's uh video if you weren't here. But really we're kind of able to break it down such that we have this attention mechanism and that is really just uh there's four linear projections that happen and then there's the actual all versus all uh attention matrix multiplication that's happening and then there was the feed forward network and that two-stage feed forward network and both of those in

1:35 terms of the number of parameters can be pretty um concisely described based on both n and E and we did that on Monday. Today I also want to talk about what

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Joseph Zambreno.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.