Training language models with reinforcement learning
Overview
Mohit Iyyer contrasts on-policy distillation, which uses dense teacher token probabilities, with reinforcement learning from a single rollout-level reward such as a calculator result or unit test. He lays out the post-training pipeline, explains why discrete sampling prevents ordinary backpropagation through generated responses, and derives the setup for REINFORCE, a policy-gradient algorithm proposed in 1992. The lecture ends by showing why naively sampling to estimate the reward gradient adds an unwanted probability factor, previewing the log-probability trick that cancels it.
Key takeaways
- On-policy distillation can backpropagate token-level reverse-KL scores only when the teacher exposes its full vocabulary probability distribution; a black-box verifier provides no such dense signal.
- A binary verifier such as a calculator or code unit test gives a clean rollout-level reward, but forces the policy to solve credit assignment without knowing which token caused failure.
- Reinforcement learning can improve beyond a teacher's demonstrated outputs because it optimizes verified task success rather than imitating a teacher distribution.
- SFT differentiates through the probability of a provided target response, whereas RL samples its own response and receives reward only after generation, making ordinary backpropagation through the rollout unavailable.
- The REINFORCE gradient uses reward-weighted changes in response probability; applying the log-probability derivative cancels the extra policy-probability factor introduced by naive sampling.
- A useful RL starting policy must already produce some positively rewarded rollouts; if every response fails, the training process has no examples of successful behavior to reinforce.
Chapters
- A student language model samples a response Y conditioned on prompt X.
- A teacher scores each generated token using its probability conditioned on the prompt and preceding response tokens.
- The per-token score is the teacher log probability minus the student log probability, derived from reverse KL divergence.
- Backpropagation updates the student using these dense token-level scores.
- The OPD update requires access to the teacher's full vocabulary distribution, illustrated with 64,000 possible word types at each position.
- Open teacher models can expose token probabilities, while services such as Claude or GPT generally do not provide their sampling distributions.
- Storing teacher and student distributions is expensive, motivating policy-gradient alternatives.
- OPD differentiates through student probabilities while treating the sampled rollout as fixed.
- For the prompt “What is 15% of 80?”, a teacher can identify the erroneous token in a response that ends with 1.2 instead of 12.
- A calculator or other verifier can instead assign a single reward, such as 1 for a correct answer and 0 for a wrong one.
- Unit tests for code and answer checks for math provide reliable outcomes without token-level explanations.
- The missing information about which generated tokens caused failure creates the credit-assignment problem.
- A binary reward does not directly identify which token or reasoning step caused an incorrect response.
- Unlike distillation, reinforcement learning is not capped by a teacher's capabilities; verified proofs and correct solutions can reward newly discovered behavior.
- The training objective is to make high-reward rollouts more probable and low-reward rollouts less probable.
- Iyyer notes that frontier-model math capabilities are developed using high-quality verification signals and carefully designed environments.
- The outlined sequence is transformer pretraining, supervised fine-tuning (SFT), optional on-policy distillation, then reinforcement learning.
- The RL policy πθ starts from the strongest available post-trained model and changes as its parameters θ are updated.
- Each RL iteration samples responses from the current policy, scores them with a reward source, and uses the score to update the policy.
- Post-training often includes repeated stages, and its work spans many tasks, unlike pretraining's strong focus on data and infrastructure.
- Verifiable rewards suit tasks with objective checks, including calculator agreement, passing code unit tests, theorem-prover validation, and legal chess moves.
- These checks give clean correctness signals but do not readily score subjective tasks such as writing a story about unicorns.
- A reward model learns to predict human judgments or preferences, often by ranking multiple responses to the same prompt.
- RLHF means reinforcement learning from human feedback; Iyyer identifies it as a standard post-training approach and focuses this lecture on verifiable rewards.
- Both verifiers and reward models return a scalar for a prompt-response pair, without a gradient that can be propagated through the scoring source.
- In the math example, the model receives one reward of 0 after producing a wrong answer, rather than a separate score for each token.
- The methods discussed apply the final reward equally to every token in the rollout.
- Learning which parts of a response matter therefore depends on many rollouts and prompts rather than direct token-level supervision.
- For a prompt X, the objective Jθ is the expected reward of responses sampled from policy πθ.
- The expectation can be written as a sum over all possible rollouts, but the number of possible token sequences makes enumeration infeasible.
- Monte Carlo sampling estimates the objective by averaging rewards across N sampled responses.
- The sampling process already reflects response probabilities, so frequent policy outputs are more likely to appear in the estimate.
- The reward may come from a black-box calculator or chess engine, so the model cannot backpropagate through the reward computation.
- Policy parameters influence the reward only by changing the distribution from which response Y is sampled.
- Token sampling makes discrete choices at every generation step, blocking ordinary backpropagation through the sampled sequence.
- This non-differentiability is the central obstacle addressed by policy-gradient methods.
- In SFT, a human or teacher supplies the target Y, and the model minimizes the negative log probability of that fixed response.
- Because the target is provided, small parameter changes smoothly alter the loss and standard backpropagation applies.
- In RL, the policy must sample tokens, complete a rollout, and only then receive reward R(X,Y).
- A small change in θ may change token probabilities without changing the sampled output, so the reward does not provide a smooth direct path for backpropagation.
- REINFORCE, proposed in 1992, is introduced as a foundational policy-gradient algorithm still used alongside newer methods.
- Its objective is to increase the probability of positively rewarded responses and decrease the probability of negatively rewarded responses.
- For simplicity, the derivation considers one sampled response for one prompt, though practical training uses many prompts and may use multiple rollouts.
- A single rollout gives a noisy, high-variance estimate, while generating language-model responses is computationally costly.
- For a sampled response, the reward is a scalar independent of θ, while the policy probability πθ(Y|X) depends on the model parameters.
- The expected-reward gradient contains the reward multiplied by the derivative of the response probability.
- Positive reward pushes toward increasing that response's probability; negative reward pushes toward decreasing it.
- The sum over all possible responses cannot be enumerated, so the key challenge is obtaining a valid sample-based gradient estimate.
- Naively averaging reward times the derivative of response probability over sampled rollouts estimates a different quantity from the desired gradient.
- The naive estimate introduces an extra factor of πθ(Y|X), which is absent from the gradient sum being approximated.
- Iyyer previews differentiating log πθ(Y|X): its derivative equals the probability derivative divided by the probability, canceling the unwanted factor.
- The full REINFORCE derivation and the corrected sampling estimator are left for the next lecture.
Transcript excerpt
0:03 All right, let's get started everyone. >> So today we are going to start our um exploration into reinforcement learning. Um and so last time and the time before that we learned quite a bit of uh background about um policies, KL divergence, um rollouts, expectations, things like that. Um, now we're going to see uh, you know, how we can incorporate all of those things into updating a language model when we only receive a score at the very end of the roll out rather than um at every token as we do in uh, on policy distillation. So before we do that, let me just
0:53 summarize um the OPD algorithm so that um we're on the same page and then we'll start to diverge away from this algorithm. So first we generate a roll out from the student given a prompt X. And so this is uh something like Y where Y is a response is drawn from our students probability distribution um conditioned on the prompt X. Um, and now that we have this roll out, right, maybe like the answer to a math problem that
1:40 this model has generated, the next step is to score the uh roll out using a teacher model, right? So we are basically computing these log probs of the teacher model of every token in the response conditioned on the prompt and every token um that occurred before that specific one in the response. Um so this is what we saw before. This scoring process happens um in parallel uh just like preill because the teacher
2:31 already knows all the future tokens in the response. The response was generated by the student model. [snorts] Um and then uh finally we uh can use the uh teachers probability distribution and the students probability distribution at
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Mohit Iyyer.