Save this video free

11/16/26: Lect-11 (cont.)

Joseph Zambreno · 1:05:45 · Watch on YouTube

Part of Iowa State EE/CPRE/CYBE 3250. See all 6 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

Joseph Zambreno explains how multi-head attention, residual connections, layer normalization, and per-token feed-forward networks fit together in Transformer blocks, then compares encoder-decoder, BERT-style encoder-only, and GPT-2-style decoder-only architectures. He derives the parameter costs, including 4D² for attention and 8D² for the feed-forward network per block, and connects model scaling to training methods such as AdamW and learning-rate warm-up.

Key takeaways

Chapters

0:00 Mini-Project Work, Exam 3, and the End-of-Semester Schedule
2:20 Why Transformers Use Multi-Head Attention
5:00 Model Width D and the Dimensions of Q, K, and V
8:00 Transformer Blocks, Residual Connections, and Layer Normalization
12:00 Original Transformer: Positional Encoding and Sequence Generation
24:00 Layer Normalization, Model Complexity, and the Per-Token FFN
28:00 Counting FFN Parameters and Understanding GELU
40:00 Encoder-Decoder, Encoder-Only, and Decoder-Only Transformers
45:00 BERT and RoBERTa: Encoder-Only Training Objectives
48:00 GPT-2's Decoder-Only Architecture and Positional Embeddings
52:00 GPT-2 Parameter Accounting: Embeddings, Attention, and FFN
57:00 How Depth and Width Drive GPT-Scale Model Sizes
1:01:00 Training Attention with Backpropagation and Scheduling AdamW

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:00 So, in interest of time, let's get started. Um, you may have noticed I haven't provided feedback yet on the mini project uh revisions. Um, I looked at every one of them and they're generally fine. I just wanted to take a little bit more time, probably this evening, to give you feedback. For some of you, the feedback was just thumbs up. So, if you submitted something and you got a thumbs up, just just get started. Um, I guess it goes without saying that you do have to put effort into it, though, right? And it's a different kind of effort than the lab effort. So for the lab effort when you're working on say lab six you you know when you're making progress. I was on section one now I'm on section two. Now I'm on section three. And and for a mini project it's always uh more tricky than that. And I know some of you set uh weekby-eek schedules and I didn't you know focus too carefully on that because

0:48 as soon as you get started usually that schedule kind of gets blown up. But but you want to put some serious time into it. You know for at least a few of those projects. I think if you just took the time that was allocated to your team in lab, I I don't think that would probably be enough. So this this is your final like experience in this particular class, I I think everyone at least the way they wrote, they want to do a good job on those projects. So it's just going to take some time and this could be a good use of your your break time to at least do more planning, more reading, more quick sort of rapid prototyping of

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Joseph Zambreno.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.