On-policy distillation (cont'd)
Overview
Mohit Iyyer reviews difficult exam questions, then explains on-policy distillation (OPD) as reverse-KL training on responses generated by a student and scored token by token by a fixed teacher. He connects the method to mode seeking, teacher-guided credit assignment, and a reported Qwen 3 8B result that rose from 60% to 70% math accuracy after 150 OPD steps, compared with an estimated 2 million SFT examples for the same accuracy.
Key takeaways
- A perplexity of 1.5 on older Wikipedia articles versus 35 on post-crawl articles is strong evidence of pretraining-data contamination, not simply easier language.
- Post-softmax causal masking does not renormalize retained attention weights, so future-token scores can affect the scale of a position's output even when future value vectors are zeroed.
- Forward KL tends to make a capacity-limited student cover teacher modes, whereas reverse KL can favor one mode and omit others.
- On-policy distillation evaluates tokens along the student's own sampled response, using teacher and student conditional distributions at every position to generate targeted updates.
- Token-level teacher feedback can pinpoint a mistake such as the incorrect 1.2 answer in the 15%-of-80 example, unlike a single binary score for the whole response.
- In the reported Qwen 3 8B math experiment, 150 OPD steps raised held-out accuracy from 60% to 70%; the comparison estimated 2 million SFT examples would be needed to reach the same score.
Chapters
- The exam median was 29 out of 40, roughly matching the previous year's result despite a somewhat harder exam.
- Mohit Iyyer sets final grade thresholds based on performance across exams, projects, and other coursework, so the thresholds can change each year.
- Students who find scan or marking errors, such as a misread answer, should submit a regrade request; makeup exams still awaiting scans may temporarily show zero.
- A language model trained on a web crawl had perplexity 1.5 on 2017 Wikipedia articles but 35 on articles written after the crawl.
- The unusually low score on older articles strongly suggests those evaluation examples were included in pretraining data.
- Benchmark results need a contamination caveat when labs do not disclose their training data; the difference is not best explained by older text being easier.
- In chat-template fine-tuning, the loss mask computes loss only on assistant response tokens, not preceding user or assistant-prefix tokens.
- The model can skip forming vocabulary-wide probability distributions and negative log-likelihoods at masked positions without changing the loss or its gradient.
- It must still compute transformer representations and attention for earlier positions because response-token hidden states depend on them; practical gains depend on implementation efficiency.
- In an exam scenario where a bug makes every attention output zero, residual connections still carry the token embedding through the transformer.
- With no positional embeddings, the trained model's next-token prediction can still depend on the current token, even though attention contributes nothing.
- Standard causal attention masks future logits before softmax, producing normalized weights over only current and earlier tokens.
- Applying the mask after softmax sets future weights to zero without renormalizing the remaining weights, so the retained attention weights may sum to less than one.
- The amount of probability removed depends on future-token scores and count, which can affect the output; Mohit Iyyer accepted that the exam wording about whether outputs 'depend on' future tokens was ambiguous and said points would be returned.
- Two labs can hold model size and total training tokens constant while using sequence lengths of 2,000 and 32,000 tokens.
- The 32,000-token setup uses more training FLOPs because full causal attention has a quadratic sequence-length cost.
- A limited student modeled as a single Gaussian may cover both modes of a two-mode teacher when optimized with forward KL.
- Reverse KL weights errors by the student's probability, encouraging mode seeking: the student can concentrate on one teacher mode and omit another.
- Pretraining and supervised fine-tuning with target examples illustrate forward-KL-style learning, while learning from student-generated responses motivates reverse KL.
- Google DeepMind proposed the original on-policy distillation algorithm in 2024; later variants include more efficient approaches associated with Thinking Machines.
- For prompt x, the student first generates response y, written as a sample from the student's conditional policy.
- A powerful teacher must expose its token probability distributions, so this setup requires access to teacher logits rather than a closed service that only returns text.
- Concatenate the original prompt with the student's generated response and run the sequence through the teacher in a parallel prefill pass.
- At each response position, compare the teacher and student next-token distributions conditioned on the prompt and the preceding student-generated tokens.
- The teacher evaluates the student's actual trajectory rather than generating a separate answer, even if the teacher would not have produced that trajectory itself.
- At position i, the reward-like signal is the teacher's log probability for the sampled token minus the student's log probability for that token.
- A positive signal means the teacher assigns the token higher probability and the student should increase its probability; a negative signal indicates the student overweights it.
- Backpropagation updates the student while keeping the teacher fixed, with the goal of reducing the distribution mismatch.
- Teacher and student generally need the same tokenizer; on-policy self-distillation can also use the same model with extra hints in the teacher's input.
- For the prompt 'What is 15 percent of 80?', a small Qwen-based student generated the incorrect calculation ending in 1.2.
- The teacher can assign near-zero discrepancy to reasonable early tokens and a strongly negative signal to the erroneous answer token.
- Unlike a binary reward for the whole response, token-level teacher scores identify where the reasoning went wrong, avoiding much of the credit-assignment problem.
- OPD is limited by its teacher: matching a fixed teacher does not generally let the student surpass that teacher's capabilities.
- A Thinking Machines experiment started with Qwen 3 8B and supervised fine-tuning on 400,000 math problems, reaching 60% accuracy on held-out problems.
- Using the Qwen 3 8B instruction-tuned model as teacher, OPD reached 70% accuracy after 150 update steps.
- The reported comparison estimated that SFT would require 2 million examples to reach 70% accuracy, and characterized OPD as 10 to 30 times cheaper.
- SFT trains against a one-hot target, while OPD uses the teacher's soft distribution to convey information about multiple plausible next tokens.
- Because OPD scores the student's own generations, it can preserve correct parts and target mistakes on the student's actual trajectory, reducing exposure-bias mismatch with human-labeled examples.
- Mohit Iyyer frames OPD as a bridge from supervised fine-tuning to reinforcement learning, which can pursue improvements beyond a fixed teacher; the next lecture will introduce reinforcement learning.
Transcript excerpt
0:02 So today, I want to spend some time going over the exam which we finally had everyone take the makeup. For those of you who took the makeup, we still haven't scored scanned some of your exams. So please don't be alarmed if it says you have like a zero. We will scan those and put them in ideally today or tomorrow. Okay, so high level exam went pretty well. The median was about 29 out of 40 which is basically what it was last year. So good job. I know people are like that seems pretty low, but it's actually fine.
0:49 Just what I did last year also was show the grade distribution from the year before. So this is how I would expect the grade distribution to look this year too in case you're panicking and you think you did poorly. Almost everyone in this class, if you show up and do your projects and the exams and the homeworks will get a good grade. So please don't panic even if you you know got below the median score. That said the exam was actually a little bit harder this time, so I'm glad to see that the scores remained pretty much the same. And so I'm going to spend maybe the first 20 minutes here going
1:37 over some of the more challenging problems that basically had like random performance across the class. But you know most of the questions went went fairly fairly well. So before I do that, any questions about like grades or things like that? Yes. >> So it sounds like >> is a distribution of the maybe applied at the end of >> Yeah. Yeah. So uh yes, that's a good question. So these thresholds are decided by me based on how the class performs on both
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Mohit Iyyer.