Save this video free

Training language models with reinforcement learning

Mohit Iyyer · 1:08:40 · Watch on YouTube

Part of University of Maryland CMSC 723. See all 11 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

Mohit Iyyer contrasts on-policy distillation, which uses dense teacher token probabilities, with reinforcement learning from a single rollout-level reward such as a calculator result or unit test. He lays out the post-training pipeline, explains why discrete sampling prevents ordinary backpropagation through generated responses, and derives the setup for REINFORCE, a policy-gradient algorithm proposed in 1992. The lecture ends by showing why naively sampling to estimate the reward gradient adds an unwanted probability factor, previewing the log-probability trick that cancels it.

Key takeaways

Chapters

0:00 On-Policy Distillation: Generate, Score, and Update
5:50 Why Dense Teacher Probabilities Enable Backpropagation
10:00 Replacing Token Scores with a Binary Math Verifier
17:00 Why Reinforcement Learning Can Exceed the Teacher
22:00 From Pretraining and SFT to the RL Policy Model
29:30 Choosing Rewards: Verifiers, Reward Models, and RLHF
36:00 One Terminal Reward Creates a Credit-Assignment Challenge
39:00 Maximize Expected Reward Across Sampled Responses
43:00 Why the Reward Gradient Cannot Pass Through Sampling
47:00 SFT Uses Fixed Targets, While RL Samples Its Own Responses
55:00 REINFORCE: A Basic Policy-Gradient Algorithm
57:00 The Reward-Weighted Probability Gradient
1:03:30 Why the Log-Probability Trick Is Needed

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:03 All right, let's get started everyone. >> So today we are going to start our um exploration into reinforcement learning. Um and so last time and the time before that we learned quite a bit of uh background about um policies, KL divergence, um rollouts, expectations, things like that. Um, now we're going to see uh, you know, how we can incorporate all of those things into updating a language model when we only receive a score at the very end of the roll out rather than um at every token as we do in uh, on policy distillation. So before we do that, let me just

0:53 summarize um the OPD algorithm so that um we're on the same page and then we'll start to diverge away from this algorithm. So first we generate a roll out from the student given a prompt X. And so this is uh something like Y where Y is a response is drawn from our students probability distribution um conditioned on the prompt X. Um, and now that we have this roll out, right, maybe like the answer to a math problem that

1:40 this model has generated, the next step is to score the uh roll out using a teacher model, right? So we are basically computing these log probs of the teacher model of every token in the response conditioned on the prompt and every token um that occurred before that specific one in the response. Um so this is what we saw before. This scoring process happens um in parallel uh just like preill because the teacher

2:31 already knows all the future tokens in the response. The response was generated by the student model. [snorts] Um and then uh finally we uh can use the uh teachers probability distribution and the students probability distribution at

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Mohit Iyyer.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.