Transformer configurations & tokenization

Overview

Mohit Iyyer compares transformer attention masks, explaining how prefix-LM, sliding-window, and document-packing masks change context access, training behavior, and inference cost. He then motivates byte-pair encoding (BPE) as a data-driven alternative to word- or character-level tokenization, showing how frequent adjacent pairs become vocabulary merges and how language imbalance affects token lengths.

Key takeaways

Prefix-LM masks give input tokens bidirectional context and output tokens causal context, but switching a causally pre-trained model to this setup can create a harmful distribution shift.
In multi-turn conversations, prefix-LM attention can require recomputing earlier KV representations when prior outputs become bidirectional context; causal attention instead allows the cache to grow incrementally.
Sliding-window attention reduces attention cost by limiting each token to nearby positions, but distant information must pass through more layers; hybrid models use occasional global-attention layers to mitigate that bottleneck.
Document-packing masks save padding computation by concatenating sequences while enforcing hard attention boundaries, so representations cannot exchange information across packed documents.
BPE creates subword coverage by repeatedly merging frequent adjacent pairs, balancing compact sequences against the unknown-token problems of word-level vocabularies.
BPE’s corpus-frequency dependence can disadvantage lower-resource languages: fewer merges mean more tokens per sentence and potentially slower generation.

Chapters

0:00 Class Agenda and Course Logistics

The lecture continues transformer configurations before moving to tokenization.
The project proposal deadline is extended to Friday while the autograder is brought online.
Homework covering pre-training and post-training is planned for release after the exam; Wednesday class and office hours are canceled.

2:16 Decoder, Encoder, and Encoder–Decoder Attention Masks

Decoder-only language models use a lower-triangular causal mask so each token attends only to prior tokens and itself.
Encoder models use unmasked, bidirectional attention across the input.
Encoder–decoder models combine a bidirectional encoder with a causal decoder connected through cross-attention.

4:00 Prefix LM Combines Input and Output in One Transformer

A prefix-LM represents an input such as a French sentence and its English translation as one concatenated sequence.
Input tokens receive bidirectional attention, while generated output tokens use causal attention.
The loss is applied to the response tokens, not the input tokens.

8:15 Why Bidirectional Input Helps Prefix-LM Translation

With a prefix-LM mask, each French input token can incorporate information from the entire sentence, including later words.
That full context can produce richer input representations for translation than a causal mask, which hides later French tokens from earlier ones.
Causal masking remains necessary over English output tokens to prevent access to future generated words.

11:51 Why Pre-Trained Language Models Keep Causal Attention

During next-token pre-training, causal attention supports a prediction loss at every sequence position.
Bidirectional input positions cannot serve the same next-token objective without exposing the target, reducing useful training predictions.
Switching a causally pre-trained model to a prefix-LM mask at fine-tuning or inference creates a distribution shift that can outweigh bidirectional-context benefits.

17:06 Prefix-LM Masks Complicate KV Caching in Conversations

Causal attention supports an incremental KV cache: new conversation tokens can be appended without recomputing earlier representations.
Under a prefix-LM mask, prior assistant responses become part of the next turn's bidirectional context, requiring earlier representations to be recomputed.
Prefix-LM masks remain useful when context is fixed, such as bidirectional image-token attention paired with causal text generation.

22:19 Sliding-Window Attention Limits Each Token’s Context

Causal attention has quadratic sequence-length cost because each query can attend to all preceding keys.
Sliding-window, or local, attention restricts each token to a fixed number of nearby positions, reducing attention computation.
A narrow window can miss long-distance dependencies, so local attention is often combined with occasional global-attention layers.

27:05 Document-Packing Masks Reduce Padding Waste

Variable-length documents in a batch often require padding to a shared length, and padded positions waste GPU computation.
Document packing concatenates multiple short documents into one sequence and uses a block mask to keep their attention separate.
Unlike overlapping sliding windows, the block mask creates hard boundaries that prevent information from flowing between documents.

31:09 Local Attention’s Effective Window Grows with Depth

With a window of W tokens, each layer can extend a representation’s receptive field by about W minus one additional positions.
After L layers, the effective context grows roughly linearly with depth, illustrated with a two-token window across four layers.
Very deep local-only models force distant information through many layers, resembling recurrent-network information-propagation bottlenecks.

39:39 Block Boundaries Stop Context Growth Across Documents

A packed-document block mask prevents a token such as “Claude” from receiving information from a separate document, even as more layers are added.
The effective window resets at each document boundary because the mask has no overlap between blocks.
Training, fine-tuning, and inference should use matching mask configurations because models learn the context available at each layer.

45:12 Tokenizer Goals: Coverage, Shared Vocabulary, and Short Sequences

Whitespace tokenization can map words to vocabulary IDs, but an out-of-vocabulary term such as “LLM” may collapse to an information-losing unknown token.
A useful multilingual tokenizer aims for shared vocabulary across languages, few unknown tokens, and short token sequences.
Shorter sequences reduce the number of predictions and attention computation; self-attention cost grows quadratically with sequence length.

53:03 Why Word-Level and Character-Level Tokenization Fall Short

Splitting on spaces and punctuation mishandles cases such as “O’Neal’s” and can turn an unfamiliar term like “GPT8” into an unknown token.
Character tokenization can represent unfamiliar words when its character inventory covers them, but produces sequences roughly four to six times longer than word-level tokenization.
Subword tokenization offers a middle ground: frequent words can remain whole while rare words break into smaller pieces, often with byte-level coverage for Unicode text.

58:43 BPE Builds a Vocabulary by Merging Frequent Pairs

Byte-pair encoding starts with characters as the vocabulary, pre-tokenizes a corpus, and counts adjacent token pairs.
In the toy corpus, “hug” appears 10 times, “pug” 5, “pun” 12, “bun” 4, and “hugs” 5; the most frequent pair is merged into a new token.
The tokenizer retokenizes after each merge and repeats until the designer’s target vocabulary size, such as 32,000 or 64,000, is reached.

1:07:07 BPE Language Imbalance and Vocabulary Finalization

Because BPE merges depend on corpus frequency, high-resource languages can receive many whole-word or subword merges while lower-resource languages remain closer to character-level tokenization.
Less efficient tokenization can make lower-resource-language outputs longer and slower to generate.
The vocabulary and its embedding rows are finalized after the target number of merges; model training then learns embeddings for those token types.

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Mohit Iyyer.