Pretraining vs post-training
Overview
Mohit Iyyer compares transformer pretraining, which uses trillions of tokens and next-token prediction to build linguistic and world knowledge, with post-training, which shapes an existing model to follow instructions using techniques such as supervised fine-tuning. He explains how attention masks and loss masks distinguish these objectives, then contrasts causal language models with BERT-style encoders and encoder-decoder transformers, before connecting quadratic attention cost to more efficient mechanisms.
Key takeaways
- Pretraining and supervised fine-tuning can use the same transformer and next-token loss, but differ in data and loss masking: pretraining learns from every token, while SFT commonly computes loss only on assistant responses.
- Causal attention masks prevent future-token leakage, whereas loss masks decide which positions train the model; the two masks serve distinct purposes.
- BERT's bidirectional attention supports masked-token prediction and strong text embeddings, but makes iterative generation inefficient because newly filled tokens change representations across the sequence.
- Encoder-decoder translation combines an unmasked source encoder with a causal target decoder; cross-attention connects decoder queries to encoder keys and values.
- Full self-attention requires quadratic compute as context length grows, while FlashAttention lowers memory use through tiling without removing that quadratic compute cost.
Chapters
- A randomly initialized transformer has random embeddings, attention weights, and feed-forward weights, so its predictions are initially unusable.
- Pretraining creates a base model with linguistic knowledge and information about the world, while post-training shapes its behavior for user-facing tasks.
- The shift from task-specific systems to models such as ChatGPT lets one model handle translation, summarization, and question answering.
- Pretraining commonly processes trillions of tokens, with web crawls such as Common Crawl supplying a large share of the raw text.
- Internet data requires filtering because it contains low-quality material; identifying useful mixtures often requires training and evaluating models.
- Textbooks and published novels are considered valuable data, and their use contributes to copyright disputes involving AI companies.
- The basic pretraining objective is next-token prediction on a large dataset, with the goal of learning language, world knowledge, and procedural knowledge.
- Mixture-of-experts models can increase parameter count while keeping per-token computation closer to the cost of activating only selected experts.
- Larger models may generalize a learned procedure, such as four-digit multiplication, to numbers not seen in training; the exact definition and causes of emergent capabilities remain debated.
- Post-training aims to make models follow arbitrary user instructions while remaining relatively harmless, and supports chat, reasoning, and tool-using behavior.
- Whether a model's apparent new capability comes from generalization or training-data exposure is difficult to establish when the corpus contains enormous volumes of text.
- Future capability growth is uncertain, especially as models collaborate with other models and interact with external environments.
- Supervised fine-tuning, or SFT, trains on prompt-response pairs, often built from human demonstrations of high-quality task performance.
- Models can generate synthetic examples for post-training, and model-generated text is also increasingly used to augment pretraining data.
- Synthetic data can add useful capabilities, but improving on a model using its own outputs raises quality-control and data-selection challenges.
- Pretraining and SFT generally use the same transformer architecture and next-token negative log-likelihood loss.
- For an SFT example such as “What is 5 + 5?” with answer “10,” training can focus its loss on the response rather than learning to reproduce the prompt.
- The principal differences are the data format and the tokens included in the loss, not a fundamentally different model architecture.
- A causal attention mask prevents each position from seeing future tokens, typically by assigning their attention scores negative infinity before softmax.
- Standard pretraining computes next-token loss at every token position, including the end-of-sequence prediction.
- An attention mask controls what information representations can access; a loss mask separately controls which positions contribute to training.
- Chat templates use special markers to distinguish user messages from assistant responses and to mark when an assistant turn ends.
- Modern SFT usually keeps the causal attention mask but masks loss on user tokens, training on assistant response tokens and the assistant end marker.
- In multi-turn conversations, the loss can be applied to multiple assistant turns while user-provided turns remain context.
- SFT still computes prompt representations and key-value information because assistant predictions must condition on the user input, even when prompt-token loss is masked.
- Training batches use GPU parallelism; packing documents into fixed-length sequences can reduce padding, but attention masks must prevent information leaking between documents.
- Ablations generally favor masking user-token loss so training emphasizes assistant capabilities, although loss on long or complex prompts can help in some data settings.
- BERT, released by Google in 2018, uses unmasked bidirectional attention, allowing every token to attend to tokens both before and after it.
- Because bidirectional context reveals the next token, BERT uses a masked-language-modeling objective: roughly 15 to 30 percent of input tokens are replaced with a special mask token.
- The loss is computed at masked positions to predict the original tokens, rather than predicting the next token at every position.
- BERT is trained to fill missing tokens using context on both sides, so it is not naturally suited to left-to-right text generation.
- Filling one end-position mask at a time does not match its training setup and requires recomputing representations as newly generated tokens change bidirectional context.
- Autoregressive transformers can reuse a key-value cache during generation because earlier token representations do not attend to future tokens.
- BERT-style encoders can produce document embeddings for clustering, semantic retrieval, and classification tasks such as sentiment analysis.
- Diffusion language models attempt parallel drafting and iterative refinement, potentially generating many tokens at once, but currently underperform standard autoregressive decoding.
- Encoder-based language models are seeing renewed use in specialized applications, including diffusion approaches and models such as JEPA.
- The original transformer formulation uses an unmasked encoder to represent a source sentence, such as French, and a causal decoder to generate its English translation.
- Decoder cross-attention uses decoder queries and encoder keys and values to select source-language information relevant to each output token.
- Training loss is computed on decoder outputs; modern systems often prefer a single decoder-only model with prompt-token loss masked instead of a separate encoder and cross-attention stack.
- Full self-attention scales quadratically with sequence length: a sequence of 4 tokens has 16 attention scores, while 8 tokens require 64.
- Feed-forward computation scales linearly with token count, so attention can become the dominant cost for sufficiently long sequences.
- FlashAttention reduces memory use by tiling computation without materializing the full attention matrix, while retaining quadratic compute; newer approaches explore alternative attention and recurrent hybrids.
Transcript excerpt
0:01 Okay, let's get started. So today we are going to um continue with our discussion of transformers but uh start talking about the different ways in which they can be applied. And so we're going to spend quite a lot of time today talking about different uh attention masks and the implications of each of these masks on what the model can and cannot model and what the um end applications of these models um are. So um just as a quick review of where we are at this point, we have um kind of finished covering the core architectural basics of a transformer. You now know the self
0:51 attention layer, the multi head aspect, the masking, um the feed forward layer that gets split into several different experts in ane. Um, so we're basically pretty up to date with the state-of-the-art in um, architectures for language models. Obviously, there's a bunch of low-level tweaks here and there, but the the general concept is um, it should be clear. Uh, so now we're going to move on to the actual training of these models and scaling them. So the theme for this class is and actually for quite a few of the lectures that follow is pre-training versus
1:38 postraining. Um and at this point it would be worthwhile to um kind of draw a diagram of the pipeline in which um the big labs train their language models like chat GBT or a cloud. Um what we've talked about so far is this process where we start with a random a randomly initialized transformer So again, this means that all of the parameters in this model, the embeddings, the WKD for all of the heads and layers, the feed forward weights for all of the
2:26 experts at all of the layers. All of
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Mohit Iyyer.