Lecture 9d
Overview
The lecture connects transformer mechanics to the ways language models represent meaning: residual connections preserve information, attention contextualizes token embeddings, and MLPs may store factual associations. It also explains image tokenization through Vision Transformers and vector-quantized autoencoders, then traces the 2017 Transformer to 2018 GPT, whose 117 million parameters were trained on BookCorpus using next-token prediction before task-specific fine-tuning.
Key takeaways
- Residual connections let transformer attention and MLP layers focus on modifying representations, while preserving a direct path for information that should remain unchanged.
- Cosine similarity compares embedding directions, and vector arithmetic can encode relationships such as king minus man plus woman landing near queen.
- Self-attention contextualizes ambiguous tokens such as mole and can combine multiple tokens into a concept not represented by any single codebook entry.
- Transformer MLPs can be interpreted as learned feature tests: weights and biases activate a ReLU, and output weights add an associated representation, such as a basketball-related vector for Michael Jordan.
- Vision Transformers tokenize 16 by 16 image patches with a learned projection, while vector-quantized autoencoders turn compressed image latents into discrete codebook tokens.
- GPT's 2018 recipe paired next-token pretraining on unlabeled text with task-specific fine-tuning; its 117 million parameters, 512-token context, and BookCorpus training set provide a historical baseline.
Chapters
- Text tokens index a learned embedding codebook, and positional embeddings are added element-wise to represent sequence order.
- Causal masking prevents a next-token transformer from seeing future characters, so each position predicts only from its preceding context.
- For binary classification, output embeddings can be averaged across positions and mapped to class scores trained with cross-entropy.
- Transformer blocks combine masked self-attention, residual skip connections, layer normalization, and position-wise MLPs.
- Residual connections, popularized in image networks such as ResNet, let information bypass a layer when its transformation is unnecessary.
- In a transformer, attention can focus on changing an embedding's meaning rather than learning to reproduce it unchanged when no change is needed.
- Residual paths around both attention and MLP sublayers make deep networks easier to train by providing routes for information and gradients.
- Layer normalization operates across features within a layer; as with batch normalization, it can make training faster and more stable.
- GloVe is a standalone embedding method, and cosine similarity compares vector directions independently of their lengths.
- Cosine similarity ranges from -1 to 1; GloVe places related words such as king, prince, and queen near one another.
- Vector differences can encode relationships: the king-to-queen difference is approximately like the man-to-woman difference.
- The analogy king minus man plus woman can land near queen, illustrating algebraic structure learned in embedding spaces.
- A single codebook embedding for mole must cover several senses, including an animal, a spy, a skin mark, and a chemistry unit.
- Self-attention uses surrounding tokens to shift an ambiguous embedding toward the sense intended by its sentence.
- Attention can also combine multiple tokens into a concept absent from the codebook, such as the phrase Eiffel Tower keychain.
- Attention weights are computed from queries and keys, then used to combine value vectors across positions and multiple heads.
- The example “Michael Phelps swims but Michael Jordan plays” illustrates how a model can associate a person with a likely continuation such as basketball.
- Attention could combine Michael and Jordan into a contextual representation; an MLP can then test that representation with learned weights and a bias.
- In the simplified example, a ReLU threshold activates when the input matches a Michael Jordan-like pattern, and the output adds a basketball-related vector.
- The first MLP matrix rows act like learned feature tests, bias terms set activation thresholds, and output weights supply the associated update.
- An embedding space of dimension D has at most D perfectly orthogonal vectors, but it can fit many more vectors that are nearly orthogonal.
- The Johnson-Lindenstrauss lemma motivates the claim that the number of approximately independent directions can grow exponentially with embedding dimension.
- For GPT-3's roughly 12,000-dimensional embedding space, the lecture gives illustrative capacity estimates of 10^8, 10^32, and 10^73 under different angular tolerances.
- These are capacity estimates, not guaranteed stored facts: the MLP weights are learned from training data, so the model can only acquire facts represented in that data.
- A Vision Transformer converts an image into a sequence by splitting it into small patches, such as 16 by 16 pixel squares.
- Each 16 by 16 patch is flattened into a 256-value vector and projected with a learned linear transformation into the model's embedding dimension.
- Once image patches and words are represented as tokens, joint training can associate visual content with language.
- The patch method is one route to multimodal learning, where a model processes images, text, and potentially other signal types.
- An autoencoder learns an encoder that compresses an input and a decoder that reconstructs it, typically with some loss of fidelity.
- For an RGB image of size H by W, the input has shape H by W by 3; an encoder can reduce spatial resolution while expanding channel dimensions.
- A vector quantizer maps latent vectors to the nearest learned codebook entries, turning continuous representations into discrete token indices.
- CLIP is cited as a well-known approach for jointly embedding images and text, while learned encoders and quantizers provide tokenization for other modalities.
- The 2017 paper Attention Is All You Need introduced a Transformer encoder-decoder architecture for machine translation.
- The encoder processes the source-language paragraph into latent representations of its content, while the decoder generates the target language one token at a time.
- Cross-attention lets target-side generation consult the encoded source meaning instead of producing an unrelated sequence.
- The decoder receives previously generated target tokens, shifted right, to predict the next token autoregressively.
- OpenAI's 2018 GPT proposal pre-trained a decoder-only Transformer on next-token prediction, making large unlabeled text collections useful for language learning.
- The approach separated general pretraining from fine-tuning for tasks such as chatbot responses, search, and sentiment classification.
- The reported model used byte-pair encoding with 50,000 tokens, 12 Transformer blocks, 12 attention heads per block, an embedding dimension of 768, and a context length of 512.
- Its 117 million parameters were trained on BookCorpus, about 7,000 unpublished fiction books and roughly one billion words, at an estimated cost of $50,000.
- The lecture closes by treating GPT's 2018 model as a baseline for comparing later language models.
- The next lecture is set to follow how model size, context length, training data, and compute increased over time.
Transcript excerpt
0:01 Okay, hope you guys had a good weekend. Uh, so we're going to get back into our discussion about transformers and large language models and all that. So, uh, to sort of summarize some of what we've seen before, we have this picture here which shows how we build this text generating transformer. [snorts] Um, in this case, we're just predicting forward one character at a time, but um, as we'll talk about more today, it's it's typically like one word at a time or, you know, perhaps even something different when you have models that use words and images and language together. But anyway, um, [clears throat] these um, these inputs, characters in this case, are converted to um, integers. Um, and each of those integers is then
0:49 used to pick out the an element within a code book of what are called embeddings. Embeddings are these vectors. So these are the vectors that correspond to those integers there. And [clears throat] um those go into the uh serial concatenation of these transformer blocks. Um, we said that in addition to those embeddings, we also would like the transformer to know which is the first and the second and the third. And so with to do that, we add these additional embeddings called um positional embeddings. And they're literally vectors of the same size as our regular embeddings that are just added element-wise like that. [clears throat]
1:36 So these go through the sets of transformers to eventually learn an output sequence that is also um just in this embedding space. And then if we want to come back to the um [clears throat] the world of characters, we first have to convert those embeddings to uh probabilities where the pro this vector this PMF vector of probabilities is telling us how probable is like the
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Prof. Schniter.