In-context learning and instruction tuning
Overview
Mohit Iyyer explains how pre-training gives a language model broad knowledge but does not reliably teach it to follow user intent, then contrasts prompt-based in-context learning with post-training that changes model parameters. Using an OLMo 3 7B base-model demo, he introduces GPT-3-style prompting, supervised fine-tuning and instruction tuning, catastrophic forgetting, reinforcement learning with human or verifiable rewards, and distillation, emphasizing that pre-training and scale support generalization to unseen tasks.
Key takeaways
- Next-token pre-training on web text does not inherently teach a model to obey user intent, because typical web pages lack explicit instruction-response structure and pre-training does not distinguish user text from assistant responses.
- In-context learning changes the prompt, not the model parameters: zero-shot gives no examples, one-shot gives one, few-shot gives several, and many-shot can use thousands.
- Instruction tuning is broad-task supervised fine-tuning: diverse instruction-response examples teach a pre-trained model the meta-task of following instructions and can improve performance on tasks excluded from training.
- Pre-training supplies capabilities that instruction tuning can redirect; a randomly initialized model trained on a limited set of tasks is unlikely to generalize to an unseen task such as division.
- Narrow SFT can cause catastrophic forgetting by damaging capabilities acquired during pre-training; mixing pre-training examples into training is one possible mitigation.
- Reinforcement learning can optimize outputs without reference answers by scoring them, using human-preference reward models or verifiable checks such as arithmetic correctness and code unit tests.
Chapters
- Mohit Iyyer introduces the post-training portion of the course and opens an OLMo 3 7B base-model demo.
- The model has 32 transformer layers, 32 attention heads, a 65,000-token context length, and training on 6 trillion tokens.
- Because it has only undergone pre-training, the model generates continuations rather than behavior specifically tuned for user requests.
- Prompts about Perplexity AI, the capital of France, arithmetic, and sports produce drifting or malformed continuations instead of focused answers.
- Common Crawl and other web text mostly contain content, not explicit user instructions paired with intended responses.
- During pre-training, loss is calculated across the sequence without a user-versus-assistant distinction, so the model learns next-token prediction rather than an instruction-following objective.
- The 2020 GPT-3 paper showed that a large pre-trained model could perform many tasks through prompting without additional training.
- Zero-shot prompts give a task description or raw question; one-shot prompts add one input-output example, while few-shot prompts include several.
- Many-shot prompting extends the same approach to thousands of demonstrations, with examples illustrating task format and output expectations.
- Chain-of-thought prompting asks a model to produce intermediate reasoning, and the capability can appear in base models before reasoning-specific post-training.
- Adding the phrase “Let's think step by step” is described as a simple zero-shot prompting strategy that improved results on math benchmarks.
- The OLMo demo’s generated math reasoning and follow-up questions suggest it has encountered similar formats, while AI-generated web content raises concerns about training on weaker models’ outputs.
- In-context learning adapts a model by adding demonstrations to its prompt, without updating its parameters.
- Top-p sampling at 0.9 can produce different reasoning and continuations for the same prompt; setting temperature to zero gives greedy decoding.
- Mohit notes that major labs increasingly release post-trained, user-facing models rather than costly pre-trained base models.
- In-context learning supplies demonstrations or shots in the prompt and avoids extra training, but can perform poorly on complex tasks.
- The effectiveness of in-context learning and instruction following improves with model scale; small models may fail despite receiving examples.
- Post-training instead updates model parameters, with supervised fine-tuning, reinforcement learning, and teacher-model distillation as major approaches.
- The training pipeline starts with a randomly initialized transformer, which pre-training on trillions of tokens turns into a base model.
- Supervised fine-tuning, often called instruction tuning when its goal is broad instruction following, produces an instruction-tuned model.
- SFT datasets are usually much smaller than pre-training corpora because high-quality prompts paired with suitable responses are comparatively expensive to collect.
- Unlike SFT, reinforcement learning does not require a ground-truth response for every prompt; it uses a score to improve the expected quality of generations.
- RLHF learns a reward model from human preferences, while verifiable rewards can directly score tasks with checkable answers.
- Math answers can be checked for correctness, and code can be evaluated with execution or unit tests; DeepSeek-R1 publicized a prominent verifiable-reward approach.
- A basic distillation method has a student imitate teacher-generated response text, for example by minimizing negative log likelihood on Claude’s outputs.
- When teacher probabilities are available, the student can learn to match the teacher’s full distribution over the vocabulary, not just its sampled text.
- On-policy RL generates and scores responses from the current model, while off-policy training reuses older generations and is generally cheaper but can use stale data.
- SFT trains on input-output pairs by minimizing negative log likelihood on target responses, using loss masking to focus learning on assistant outputs.
- Repeatedly fine-tuning a pre-trained model on a narrow task such as addition can damage its broader knowledge and abilities.
- This loss of previously learned capabilities is called catastrophic forgetting; mixing in pre-training data was one mitigation used before the ChatGPT era.
- Instruction tuning is a specific form of SFT whose dataset spans many plausible user tasks rather than specializing in one task.
- Example tasks include arithmetic, solving homework, and editing an email; the model learns to follow varied instructions through their input-output pairs.
- The goal is for the meta-task, following an instruction, to generalize beyond the exact examples in the tuning set.
- A randomly initialized model trained only on labeled instruction examples is unlikely to understand language well enough to generalize to entirely new tasks.
- Pre-training supplies knowledge and capabilities, such as language structure and facts, that instruction tuning can adapt to new response formats.
- The 2021 Google FLAN work fine-tuned a pre-trained model on hundreds of NLP tasks and found improved performance on held-out tasks.
- Larger models trained on more data tend to generalize more strongly to unseen tasks after instruction tuning.
- Mohit previews upcoming coverage of instruction-tuning datasets, efficient SFT methods, forgetting mitigation, and reinforcement learning.
- The planned homework sequence has students pre-train a tiny transformer and later post-train it on a toy task.
Transcript excerpt
0:00 Okay, let's get started everyone. So, today we are going to be covering or starting the post training half of the class. So, we've talked about pre-training up till the first exam and the rest of this class will focus on post training as well as agents and applications of language models. Um All right. Let's get started. So, first I wanted to cover some of the kind of motivations behind this pre and post training split and why it is that model developers started doing post training in the first place.
0:50 So, I made this little well, I should say Claude made this little demo that serves a base model. So, we can see what this is. This is the Olmo 3 7 billion parameter base model. It was released When was it released? November Sorry, October of 2025. So, it's about 1 year old at this point. We don't really get many of these open base models released anymore. So, this is probably one of the most recent ones that that we have. Base here means that this model has just undergone pre-training. So, there's no supervised fine-tuning or reinforcement
1:39 learning, which are all things that we will cover in the upcoming lectures. So, let's just take a look at some of the things that now everyone here is equipped to hopefully understand. We see here that the model was trained on 6 trillion tokens, which seems large. It definitely is large. However, the frontier models are pre-trained on way more than that. It's got 32 layers. So, remember here each layer is like a transformer block. So, we have the self-attention, the multiple heads, and then the feedforward layer. We've got different heads. So, here 32 heads and
2:28 the context length is the length of the sequence that goes into
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Mohit Iyyer.