Lecture 9c

Overview

Transformers build context-sensitive token representations through stacked attention blocks, then use those representations for next-token generation and tasks such as sentiment classification. The lecture derives query-key-value attention, multi-head processing, positional embeddings, and causal masking, then connects them to a character-level Shakespeare demo that reached 55% test accuracy and to practical LLM choices such as subword tokenization and temperature-controlled sampling.

Key takeaways

Self-attention forms each token’s updated representation as a weighted combination of values, with softmax weights determined by query-key similarity.
Multi-head attention provides several parallel relationship patterns by splitting query, key, and value projections into smaller heads and combining their outputs.
Position embeddings are necessary because attention alone is permutation equivariant and cannot distinguish order-sensitive sentences such as “dog bit man” and “man bit dog.”
Causal masking sets future attention scores to negative infinity before softmax, preventing a next-token model from seeing the answers it is meant to predict.
A small Shakespeare character model with 64-character context, four heads, and four blocks reached 55% test accuracy, exceeding the lecture’s MLP and bigram baselines of 45% and 27%.
Subword tokenization methods such as Byte Pair Encoding and WordPiece reuse roots and suffixes, while temperature controls how concentrated or varied next-token sampling is.

Chapters

0:00 Transformer Pipeline: From Token Embeddings to Autoregressive Text

Each input token starts as a codebook embedding, then passes through stacked transformer blocks that make its representation context-dependent.
A final matrix projection and softmax produce a probability distribution over the token vocabulary.
Autoregressive generation samples one next token at a time, appends it to the context, and repeats; a sharper probability distribution generally means less randomness.

9:54 Watermarking Text Through Model-Specific Sampling Choices

A generator can watermark output by restricting which high-probability characters it samples at particular steps.
The text can remain natural-looking while a detector checks for a recognizable pattern unlikely to occur by chance.
A model owner could use this signal to investigate whether text came from a particular model, including in academic-integrity or legal settings.

13:32 Basic Self-Attention Uses Softmax-Normalized Similarity

In the simplified attention model, each output vector is a weighted sum of all input vectors, with weights produced by applying softmax to pairwise inner products.
Inner products measure alignment: positive large values indicate similar directions, values near zero indicate orthogonality, and negative values indicate opposing directions.
Because this formulation has no position information, shuffling the inputs only shuffles the outputs, a property called permutation equivariance.

17:00 Query, Key, and Value Projections Enable Multi-Head Attention

Learned matrices project each input into a query, key, and value: queries seek matches, keys provide matching information, and values supply content to combine.
Scaled inner products between queries and keys determine attention weights; scaling by vector dimension helps keep softmax weights from becoming too extreme.
Multi-head self-attention splits projections into several smaller heads, allowing different relationship patterns to be learned in parallel before their outputs are combined.

22:50 Matrix Form of Attention and Head Selection

Stacking token vectors as rows lets queries, keys, values, and attention scores be computed with matrix multiplications.
The attention function applies row-wise softmax to query-key scores and uses the resulting weights to combine value vectors.
Selection matrices can extract the relevant projection rows for each head, whose outputs are concatenated and passed through an output matrix.

26:03 Transformer Blocks Combine Attention, Layer Norm, and MLPs

A transformer block adds a skip connection around multi-head attention, so the input representation can pass through even when the attention contribution is zero.
Layer normalization standardizes features within an embedding, then applies learned scale and shift parameters; unlike batch normalization, it does not average across the batch.
A two-layer MLP processes each token embedding independently, commonly using ReLU in early models or GELU in newer ones; multiple blocks are stacked for deeper processing.

30:38 Sentiment Classification Requires Positional Embeddings

For movie-review sentiment classification, the transformer outputs can be averaged into one vector and passed to a linear classifier with one logit per category.
Multinomial logistic regression with softmax loss trains the classifier from reviews paired with labels such as one-to-five-star ratings.
Without position information, averaging makes “dog bit man” indistinguishable from “man bit dog”; adding a position vector to each token embedding preserves word order.
Position vectors can be learned from data or designed as sinusoidal encodings with alternating sine and cosine components.

37:12 Causal Masking Prevents Future-Token Leakage

Next-token training shifts a sequence such as “H E L L” to targets “E L L O,” creating predictions at multiple context lengths.
Without a causal mask, a model could copy future input tokens to the outputs instead of learning genuine next-token prediction.
The mask inserts negative infinity into forbidden future attention scores before softmax, making their weights zero so position i can attend only to positions 1 through i.

42:37 Shakespeare Character Model: 55% Test Accuracy

The character-level demo used a context length of 64, 64-dimensional embeddings, four attention heads, and four transformer blocks.
Training used positional embeddings, causal masking, dropout of 0.1, AdamW, batch size 128, learning rate 0.001, and two epochs; the run took about an hour on an older MacBook.
The model reached 55% test accuracy, compared with 45% for a two-layer MLP and 27% for a bigram model, and generated more Shakespeare-like names and phrases.
A larger example used context length 256, 384-dimensional embeddings, six heads, and six layers, training in about 15 minutes on an NVIDIA A100 GPU.

49:04 LLM Tokenization Uses Subwords Instead of Whole Dictionaries

Large language models commonly tokenize text into subwords rather than assigning a separate ID to every complete dictionary word.
Byte Pair Encoding and WordPiece can split “walking” into “walk” and “ing,” reducing vocabulary size while reusing roots and suffixes.
SentencePiece is one option for languages and text where spaces do not reliably separate words.

51:42 Token Embeddings, Temperature, and Candidate Filtering

A learned embedding matrix has shape D by N, where D is the vector dimension and N is vocabulary size; a one-hot token vector selects its embedding column.
De-embedding projects an output vector into vocabulary scores, divides by a temperature, and applies softmax; lower temperatures concentrate probability while higher temperatures spread it.
The output projection can share the transpose of the embedding matrix to reduce parameters, and generation can exclude the low-probability tail by sampling only a small set of likely tokens.

Transcript excerpt

0:01 Okay, welcome back everyone. So, we're going to continue our discussion of transformers [clears throat] uh and we're going to get a little bit into large language models today, I believe. So, >> [clears throat] >> we've been kind of using as a model problem this idea of um token prediction, predicting one new character forward in a sequence or one new word forward in a sentence, things like that. >> [clears throat] >> And I mentioned that there's different um neural network architectures that have been proposed over the decades to do that. But, they all have um some trouble looking back in time and figuring out what's really important in the past to predict the future. And so, um the real solution to that which has

0:50 kind of um caused uh this in some sense revolution um you know, this AI revolution, these chatbots, all the stuff that we're experiencing these days is based on a particular [clears throat] neural network architecture called the transformer and that is based on a particular uh mechanism or concept called attention. And so, [clears throat] we first started talking about this in the context of this phrase and trying to predict one uh character in the you know, to the right. And [clears throat] we said in order to really predict that character well you have to understand that um this word run is is a verb and dog is a subject in order to conjugate this properly because dog

1:38 is singular, you need to add a plural S to run and so on. And so, really in order to do that well um that prediction you have to sort of analyze all the different um different characters in that sentence and you have to figure out which of them matter, which of them

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Prof. Schniter.