1786 lecture9 Nov10
Overview
Jonathan Rose explains how transformer language models learn to predict the next token from large text corpora, then develops the mechanics of self-attention: contextual embeddings, dot-product scores, softmax weights, and learned query, key, and value projections. He connects multi-head attention to residual connections, feed-forward networks, parallel computation, and KV caching, then previews Assignment 4, which uses Andrej Karpathy’s minGPT and GPT-2.
Key takeaways
- Next-token prediction supplies its own training labels: each observed token becomes the target for the preceding text, allowing language models to learn from large corpora without manually labeled examples.
- Self-attention computes context-dependent representations by comparing queries with keys, applying a softmax to the scores, and using the resulting weights to combine value vectors.
- Query, key, and value projections provide trainable parameters for attention while preserving compatible representation dimensions; scaling dot products by the square root of D helps control score magnitudes.
- Residual connections preserve a direct path for information and gradients, while the feed-forward MLP typically expands each representation from D to about 4D before projecting it back.
- Transformer token positions can be processed in parallel on GPUs, and KV caching reuses prior keys and values during generation at the cost of substantial memory.
- Assignment 4 applies transformer concepts with minGPT, a 3-block model with D=48 and 3 heads, sentiment classification, text generation, and GPT-2 fine-tuning.
Chapters
- Jonathan Rose says project teams must submit a progress report with a strict word limit to keep work moving before the end of term.
- The circus assignment is due Tuesday, November 17 at 9:00 p.m., correcting an earlier reference to Monday.
- Every team member must commit their own code to their individual repository; teammates cannot make commits on someone else’s behalf.
- Assignment 4 concerns transformer internals and draws on this lecture and nearby lectures.
- A language model estimates probabilities for every vocabulary token as the next token given the preceding communication.
- Training needs no human-labeled answers: in a sentence such as “The smooth blue lake became choppy in the wind,” the next word supplies the label.
- Repeated next-token prediction across massive text collections, large models, and substantial compute produced unexpectedly capable behavior.
- Rose identifies next-token prediction as the central idea to explain to non-specialists, rather than beginning with attention terminology.
- Tokens become embeddings, which pass through repeated transformer blocks while retaining the same overall representation size.
- Models can scale by adding blocks, increasing embedding dimension D, or expanding the context length n, which Rose says grew from roughly 512 tokens to as much as a million.
- For next-token prediction, the model uses the output corresponding to the final input token, not every output position at once.
- Keeping many token representations available contrasts with recurrent neural networks, which compress prior context into a narrow state.
- A block includes multi-head self-attention, layer normalization, a feed-forward network or MLP, and another normalization step.
- Residual or skip connections add earlier representations back into later computations.
- The feed-forward network contains most of a block’s parameters, and its weights are shared across token positions within that block.
- Attention makes a token’s output depend on preceding tokens, while the feed-forward network operates on each token representation individually.
- Self-attention aims to adjust an embedding using context so that an ambiguous token reflects its meaning in that sentence.
- “Bank” can mean a river edge, a financial institution, or relying on an outcome, depending on nearby words.
- Rose also uses the pronoun “it” to illustrate how context resolves what a word refers to.
- Attention builds contextual embeddings during transformer computation rather than retrieving a separate precomputed embedding for each context.
- For “he emptied his bank account,” the example focuses on the embedding for “bank” and compares it with earlier tokens and itself.
- The comparison estimates how relevant each available token is to the token currently being processed.
- The model uses those relevance values to mix information into a revised representation of the focus token.
- Including the token itself helps preserve its original information while context is incorporated.
- Rose reviews dot product as a basic way to turn two embeddings into a single similarity score.
- For focus token xᵢ and context token xⱼ, the unnormalized score is xᵢ · xⱼ.
- A larger score gives a token more influence in the weighted context mixture.
- This first intuitive formulation compares the embeddings directly, before introducing learned projection matrices.
- Attention scores are normalized into positive weights that sum to one, allowing them to act as proportions.
- Softmax exponentiates the scores and divides each result by the sum of exponentiated scores across the available prefix.
- Because of exponentiation, softmax emphasizes larger scores more strongly than simple linear normalization.
- For position i, the causal calculation considers positions j less than or equal to i, not future tokens.
- The initial attention output is a weighted sum of token representations: each context vector is multiplied by its attention weight and added to the others.
- For the “bank” example, relevant prior tokens contribute more to the new representation than unrelated ones.
- The result is intended to preserve the focus token while folding in useful information from its context.
- This weighted-sum version is an intuition for attention; learned projections and other block operations still need to be added.
- A direct weighted sum of input embeddings contains no trainable parameters, so Rose introduces learned weight matrices.
- Multiplying a size-D vector by a D-by-D matrix produces another size-D vector while adding parameters that gradient descent can adjust.
- The matrices are randomly initialized and trained by backpropagation to improve next-token predictions.
- Rose compares the idea of learned matrices to CNN kernels, which can acquire useful patterns during training.
- Each input embedding is projected in three learned ways: query, key, and value, using separate matrices WQ, WK, and WV.
- The query represents the token seeking relevant context; keys represent the tokens being compared against it.
- Values are the representations that get combined after attention weights are computed.
- The terminology can be understood as a search: compare a query against keys, then use the resulting weights to aggregate values.
- The attention weight is a softmax over query-key dot products, divided by the square root of D to control score scale.
- The output is a weighted sum of value vectors, rather than a weighted sum of the original embeddings.
- Multi-head attention uses H separately learned sets of query, key, and value matrices.
- One common size strategy gives each head dimensions D/H, keeping the combined representation at size D; another uses a learned projection to reduce a larger concatenated output.
- Skip connections add the input representation to a block’s transformed output, helping gradients flow through deep networks.
- Rose describes residual paths as insurance against failed optimization: even if a transformation performs poorly, the original representation remains available.
- He argues that these paths also help explain why representations change gradually across transformer layers.
- Layer normalization controls activation scale, while the exact future design of language models may move beyond today’s transformer architecture.
- A transformer’s feed-forward MLP is typically two layers: it expands D to about 4D, then projects back to D.
- Rose notes that a large share of transformer parameters, and possibly many stored facts, are associated with the feed-forward network rather than attention.
- Unlike recurrent neural networks, transformer operations for multiple token positions can run in parallel, supporting efficient GPU computation.
- During autoregressive generation, previously computed keys and values can be cached; the KV cache saves repeated work but consumes substantial memory.
- Assignment 4 uses Andrej Karpathy’s minGPT code and a small model configured with three transformer blocks, embedding size D=48, and three attention heads.
- Students inspect the implementation, train on datasets from Assignment 1, replace the language-model head for sentiment classification, and experiment with text generation.
- The assignment also fine-tunes a GPT-2 model; the next lecture covers decoding because always choosing the highest-probability token does not work well.
- Rose explains that positional embeddings encode token order, while next-token training may learn some ordering information even without them; masked-word prediction fails without positional encoding.
Transcript excerpt
0:03 Okay, welcome to ECE 1786 lecture number nine. I have written a lot on the board because I'm reprising the two things I the two big pictures I had last day and I needed it needed it there. They're in the notes so I don't know if you have to write them all down but let me tell you the work that I'm I've put you put you towards in this course. two big things. Uh the project which are largely underway and people are thinking carefully about you need to submit a a progress report. In my experience as a teacher, you need to have these kind of milestones where people you have some impetus towards doing the work and not leaving it all towards the end. So there's a progress report that has very specific specifications including a hard word limit like usual. Uh in the circus assignment, which I just released, I
0:50 realized I didn't I should have put it out a little earlier. tells you what you need to deliver next week. Uh, and I also noticed that I would have normally made the due on the Monday at 9:00 p.m., but I've been saying it's November 17th, so it's actually due on Tuesday, November 17th at 9:00 p.m. Uh, um, and the other key point is I've made repositories for everyone, and I expect to see code in those repositories submitted by each member of the team. You're not allowed to let your teammate do the commits for you. you must commit your own work so that I can see that you did some work. Um, I did release belatedly assignment 4. I was hoping to change it but uh didn't. It's still the same as previously. Uh, which is about
1:39 the subject of the previous lecture in
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Jonathan Rose.