Save this video free

On-policy distillation (cont'd)

Mohit Iyyer · 1:09:25 · Watch on YouTube

Part of University of Maryland CMSC 723. See all 11 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

Mohit Iyyer reviews difficult exam questions, then explains on-policy distillation (OPD) as reverse-KL training on responses generated by a student and scored token by token by a fixed teacher. He connects the method to mode seeking, teacher-guided credit assignment, and a reported Qwen 3 8B result that rose from 60% to 70% math accuracy after 150 OPD steps, compared with an estimated 2 million SFT examples for the same accuracy.

Key takeaways

Chapters

0:00 Exam Scores, Grading Thresholds, and Regrade Requests
4:00 Extremely Low Perplexity Signals Training-Data Contamination
8:00 Loss Masks Can Skip Unneeded Output-Head Computations
11:30 Residual Connections Preserve Current-Token Information
13:00 Why Applying a Causal Mask After Softmax Leaks Future Information
19:00 Longer Training Sequences Increase Attention Compute
21:30 Forward KL Covers Teacher Modes, Reverse KL Seeks Them
32:30 OPD Samples Student Rollouts and Scores Them with a Teacher
34:00 Teacher Scoring Uses the Student's Prefix at Every Token
43:48 Token-Level Log-Probability Differences Drive Student Updates
52:00 A Math Error Shows OPD's Token-Level Credit Assignment
1:03:00 Qwen 3 8B Results: 70% Math Accuracy After 150 OPD Steps
1:07:00 Why OPD Can Be More Efficient Than SFT

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:02 So today, I want to spend some time going over the exam which we finally had everyone take the makeup. For those of you who took the makeup, we still haven't scored scanned some of your exams. So please don't be alarmed if it says you have like a zero. We will scan those and put them in ideally today or tomorrow. Okay, so high level exam went pretty well. The median was about 29 out of 40 which is basically what it was last year. So good job. I know people are like that seems pretty low, but it's actually fine.

0:49 Just what I did last year also was show the grade distribution from the year before. So this is how I would expect the grade distribution to look this year too in case you're panicking and you think you did poorly. Almost everyone in this class, if you show up and do your projects and the exams and the homeworks will get a good grade. So please don't panic even if you you know got below the median score. That said the exam was actually a little bit harder this time, so I'm glad to see that the scores remained pretty much the same. And so I'm going to spend maybe the first 20 minutes here going

1:37 over some of the more challenging problems that basically had like random performance across the class. But you know most of the questions went went fairly fairly well. So before I do that, any questions about like grades or things like that? Yes. >> So it sounds like >> is a distribution of the maybe applied at the end of >> Yeah. Yeah. So uh yes, that's a good question. So these thresholds are decided by me based on how the class performs on both

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Mohit Iyyer.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.