Save this video free

Transformer language models

Mohit Iyyer · 1:10:56 · Watch on YouTube

Part of University of Maryland CMSC 723. See all 12 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

Mohit Iyyer explains how causal self-attention scales into a Transformer through multi-head attention, stacked layers, positional strategies, and residual connections. He also shows how causal masks can expose position information, why every layer must remain masked to prevent future-token leakage, and how prefill builds a reusable key-value cache for autoregressive decoding.

Key takeaways

Chapters

0:00 From Single-Head Attention to Parallel Matrix Computation
7:42 GPT-2-Style Absolute Position Embeddings and Their Length Limit
13:14 RoPE and the Shift Toward No Positional Encoding
16:22 How a Causal Mask Can Reveal Sequence Position
24:05 Position Encodings as an Inductive Bias
31:20 Multi-Head Attention Gives the Model Multiple Views
38:50 Concatenating Attention Heads into a Token Representation
45:57 Stacking Transformer Layers with New Parameters
51:52 Final-Layer Hidden States and Next-Token Prediction
54:15 Why Causal Masking Must Continue Through Every Layer
57:21 Layer Normalization, Scaled Dot Products, and Residual Paths
1:03:48 Autoregressive Inference Starts from a User Prompt
1:06:15 Prefill Builds the Transformer KV Cache
1:09:03 KV Cache Growth and the Next Scaling Topics

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:17 All right. So today we are going to be continuing where we left off from um last time where we tal talking about the attention mechanism um and moving forward to kind of incorporating that into a much larger scale model the transformer which features a couple of different things that extend what we were talking about last time. So we talked about the uh attention mechanism. Last time we actually talked about a single head attention mechanism and today we will expand that to uh multi-head self attention where each head can um learn to focus on a different property of the prefix. Um that's the high level intuition. We will also talk about

1:07 stacking multiple of these multi head self attention layers on top of each other before you reach the softmax. Um this allows you to express more complex relationships between the uh input prefix and the output especially as we um aim to scale up our model to potentially like trillions of uh training tokens. Um we want to scale up our model um in the same way so that it has more parameters. We talked about the parameters of the self attention um language model as having the WQ W K WV weight matrices for um the single head that we talked about. And now we'll see

1:58 that we can just multiply the number of parameters dramatically by adding more heads and adding more layers. Um so that's kind of where we're going today. Um there are also a couple of concepts um uh after we cover the um kind of multi head aspect that um I wanted to emphasize regarding what we do at test time when we're decoding from the model. Um there's uh as we talked about last time a difference in behavior right at

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Mohit Iyyer.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.