SFT and LoRA
Overview
Mohit Iyyer surveys supervised fine-tuning data, including Tulu 3 and FLAN, and explains how synthetic responses from teacher models can transfer capabilities to smaller students. He contrasts the LIMA paper's 1,000-example superficial alignment hypothesis with evidence that SFT can add facts but risks catastrophic forgetting, then derives LoRA as a memory-efficient alternative that trains low-rank adapters instead of updating every model weight.
Key takeaways
- Instruction-tuning datasets such as Tulu 3 combine diverse tasks, from code generation and translation to structured tables, safety refusals, and synthetic reasoning traces, to help models generalize to unseen instructions.
- LIMA's 2023 study found that 1,000 carefully selected examples could produce a strong instruction-tuned LLaMA model, supporting the hypothesis that pretraining provides most capabilities while SFT shapes response behavior.
- SFT can encode new facts, but repeatedly training on narrow examples risks catastrophic forgetting and is less efficient for broad knowledge acquisition than pretraining.
- Vanilla distillation trains a student on teacher-generated responses using standard SFT, while richer methods can transfer probability distributions, chains of thought, attention scores, or hidden states.
- For a D by D weight matrix, full fine-tuning updates D² parameters, while rank-R LoRA trains 2DR; at D = 4,096 and R = 8, that is about 0.4 percent of the parameters.
- LoRA adapters are compact enough to share independently from pretrained model weights, and QLoRA combines adapters with quantization to lower fine-tuning resource requirements further.
Chapters
- Mohit Iyyer previews SFT datasets, distillation, and parameter-efficient fine-tuning as the lecture's main topics.
- Homework 1 asks students to implement transformer forward propagation and train on a toy task, with a Gradescope autograder.
- The stated target is 90 to 95 percent test accuracy for full credit, with unlimited submission attempts.
- Tulu 3 compiles existing instruction-tuning and SFT benchmarks into millions of instruction, input, and response examples.
- Examples range from generating Terraform HCL for an AWS auto-scaling group to writing iambic pentameter under specific constraints.
- Producing high-quality, constraint-heavy instructions and answers at this scale makes data curation and quality control difficult.
- FLAN converts traditional task-specific datasets, such as natural-language inference and translation, into instruction-following examples.
- Examples include judging whether a hypothesis follows from a premise and translating an English sentence into Gujarati.
- SFT uses loss masking to train on response tokens while excluding the instruction and input tokens.
- Safety behavior can also appear in SFT data, including examples where the model refuses requests for harmful or restricted information.
- Some Tulu examples pair word problems with long chains of reasoning generated by a stronger language model rather than a human.
- Distillation can give smaller models, such as 7-billion-parameter models, examples of capabilities that are expensive to elicit from human annotators at scale.
- Teacher-generated reasoning may contain errors, so verification matters, although partially correct traces can still provide useful training signal.
- Math and coding are especially prominent in open-model instruction tuning because their answers are often objectively verifiable.
- A coding example trains a model to write a Python function that totals the areas of rectangles from their widths and heights.
- Coding assistants combine instruction-following SFT with pretrained knowledge from large code collections, including GitHub crawls.
- Tulu includes refusal examples, such as declining to provide detailed defense strategies, alongside benign fictional prompts.
- For reinforcement learning, the prompt can be reused while the ground-truth answer is discarded; generated responses are instead scored by a reward function.
- Downstream tasks are practical applications such as question answering, summarization, and coding, distinct from next-token pretraining on general text.
- In-context learning supplies task examples in a prompt without changing model parameters.
- Instruction tuning applies SFT to examples like those in Tulu and FLAN to improve a pretrained model's assistant behavior.
- A central research question is whether SFT teaches new capabilities or mainly changes how existing capabilities are expressed.
- The 2023 LIMA paper, titled Less Is More for Alignment, tests how much instruction data is needed to elicit pretrained model capabilities.
- Its dataset contains exactly 1,000 examples from sources including Stack Exchange, wikiHow, writing prompts, and manually authored instructions.
- The superficial alignment hypothesis argues that pretraining supplies most knowledge and capabilities, while alignment teaches suitable response formats.
- LIMA's LLaMA model compared favorably with some larger instruction-tuned models in human preference tests, despite using a very small dataset.
- Later large-scale datasets such as Tulu 3 suggest that adding SFT examples can improve performance, although gains eventually plateau when a base model lacks underlying capability.
- Reasoning-focused reinforcement learning has been reported to improve performance as models produce longer chains of thought, complicating the idea that alignment only changes formatting.
- SFT can teach a new fact, such as the capital of a newly established country, by repeatedly training on a relevant completion.
- Repeated updates on narrow facts can cause catastrophic forgetting, and SFT is generally less efficient for learning facts than pretraining.
- Vanilla distillation uses a strong teacher language model to generate synthetic responses for an instruction-following dataset.
- A pretrained student, often an open-weight model such as Qwen or Llama, is then trained on those responses using ordinary SFT and response-only loss masking.
- Human-authored instructions and carefully filtered synthetic data remain important for frontier-model training; coding examples can be checked with unit tests.
- Distillation can help a smaller student gain capabilities that would be costly to acquire from a limited set of human-labeled examples.
- Distillation has prompted disputes between frontier-model companies and open-model developers over using outputs from models such as GPT and Claude to train students.
- Research into detecting model distillation and output watermarking raises security questions, including whether marks survive training on generated data.
- When available, chains of thought can be distilled alongside polished responses; raw reasoning traces may be less coherent or switch languages.
- Other approaches transfer teacher probability distributions, attention scores, or hidden states, but these require richer access than a single generated answer and may depend on architectural compatibility.
- Full SFT requires gradients and forward and backward computation for model weights, creating substantial memory demands for large pretrained models.
- The lecture motivates parameter-efficient fine-tuning as a way to make adaptation feasible on more modest hardware.
- If alignment mainly changes response behavior rather than model knowledge, freezing most pretrained parameters may preserve capability while reducing training cost.
- LoRA, or low-rank adaptation, is introduced as the dominant parameter-efficient approach, alongside alternatives such as prompt tuning and prefix tuning.
- LoRA freezes pretrained weights W and represents the fine-tuning update as BA, where B has dimensions D by R and A has dimensions R by D.
- For a linear layer, the adapted computation is h = Wx + BAx; a small rank such as 4, 8, or 16 drastically reduces trainable parameters.
- With D = 4,096 and R = 8, LoRA trains 65,536 parameters for a matrix instead of about 16.8 million, roughly 0.4 percent as many.
- LoRA adapters can be shared separately from the base checkpoint and added at inference; QLoRA further reduces resource needs by combining low-rank adaptation with quantization.
Transcript excerpt
0:05 All right. Uh sorry about that. So, we finally got this working. All right, let's get started um everyone. Um sorry for the delay. All right, so today I uh wanted to continue where we left off last uh time on supervised fine-tuning. We're going to talk about um like data sets that are commonly used for SFT at scale. Uh talk a bit about distillation and then focus on uh parameter efficient tuning methods. Um So, before we do anything, um oh, I should also mention uh we are done with homework one and will release it either today or tomorrow morning. Um it'll be due 2 weeks from
0:53 now. Um it's pretty straightforward uh implement uh like the uh forward uh prop phase of a transformer um and train it on a toy task. Um this will have an autograder on GradeScope. So, basically, you're graded on the quality of your implementation. If your transformer reaches like 90 or 95% um accuracy on the test set, um then you get full credit. Otherwise, you get no credit. So, you're basically just um uh it's not that scary. Uh you you will be able to um get the credit pretty easily, um but uh you have like, you know, unlimited tries to to do this. So, um don't worry about it.
1:43 Okay. Uh so, before I start with the iPad, I wanted to talk about um the SFT data mix that we see um in modern models. And so, if you remember from last time we talked about instruction tuning, right? Where the goal is train the model on as many different instructions as possible with the hope that at test time the model will be able to generalize to new instructions. So, I wanted to show you you know what some of these instructions
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Mohit Iyyer.