REINFORCE (cont'd)
Overview
Mohit Iyyer derives REINFORCE for language-model post-training, showing how the score-function identity turns an intractable expected-reward gradient into the sample-based estimator E[R∇θ log πθ(Y|X)]. The resulting loss is negative reward times rollout log-probability; Iyyer connects it to weighted SFT, explains sparse-reward and token-level credit-assignment problems, then introduces reward baselines and GRPO’s group-wise reward normalization.
Key takeaways
- REINFORCE estimates the expected-reward gradient with sampled responses using R(X,Y)∇θ log πθ(Y|X), avoiding direct differentiation through discrete token sampling.
- The practical REINFORCE loss is the negative mean of reward times rollout log-probability, making policy-gradient training resemble reward-weighted SFT on model-generated responses.
- With a 0/1 reward, successful rollouts receive an SFT-like likelihood update, while failed rollouts receive no update unless rewards are centered or otherwise assigned negative values.
- A scalar baseline turns reward into an advantage, allowing above-baseline responses to be reinforced and below-baseline responses to be suppressed without changing the expected gradient.
- GRPO samples multiple responses per prompt and standardizes their rewards within the group; groups with uniformly successful or uniformly failed responses have zero centered advantages and produce no update.
- A single response-level reward is applied across all generated tokens, so long trajectories create a difficult credit-assignment problem and can make prompt difficulty and rollout length consequential.
Chapters
- The upcoming Google Colab homework uses GRPO to train a small base model on keyword-and-length constraints, such as writing an eight-word sentence containing a specified word.
- The exercise emphasizes experiments and analysis because reinforcement-learning performance is sensitive to hyperparameter choices.
- A prompt X goes into the trainable policy πθ, which generates a complete response Y and receives one scalar reward at the end.
- Rewards may be binary or continuous, and verifiable examples include checking a math answer with a calculator or generated code with unit tests.
- The ordering of supervised fine-tuning, online policy distillation, and RL can vary across labs; it is not a universally fixed pipeline.
- The objective sums reward over every possible response, an intractable set of rollouts under a language model.
- Sampling is non-differentiable: changing token probabilities does not provide a direct gradient through the discrete response-generation decisions.
- A naive sample average of the differentiated rollout probabilities misses the policy-probability weighting needed to estimate the desired gradient.
- Iyyer multiplies and divides by πθ(Y|X), converting a sum over responses into an expectation over samples drawn from the policy.
- The chain rule gives ∇θ log πθ(Y|X) = ∇θπθ(Y|X) / πθ(Y|X), canceling the added probability denominator.
- The resulting estimator is E[R(X,Y)∇θ log πθ(Y|X)], so sampled rollouts can estimate the expected-reward gradient.
- πθ(Y|X) is the probability of the full response given its prompt, with θ representing the language model’s parameters.
- The sequence probability factors into conditional token probabilities, such as πθ(y₁|X), πθ(y₂|X,y₁), and subsequent next-token terms.
- Other checkpoints, including base, SFT, or teacher models, can also assign sequence probabilities, but πθ is the policy being optimized.
- Maximizing expected reward is implemented as gradient descent on a negative objective.
- For sampled rollouts, the loss is negative reward multiplied by the response log-probability, averaged across samples.
- In PyTorch-style code, compute rollout log-probabilities, multiply by corresponding rewards, take the negative mean, and call backward; rollout generation is a major practical cost.
- Pretraining and SFT minimize the negative log-probability of observed target text, while REINFORCE applies the same log-probability structure to responses sampled from the policy.
- The reward weights each sampled response’s log-probability, making REINFORCE resemble weighted SFT rather than training on human or teacher responses directly.
- A judge model can provide a graded score, such as 0 to 100, to guide a student model’s outputs.
- The response log-probability is a sum of token-level log-probabilities, but REINFORCE applies the same rollout reward to all tokens in that response.
- Long outputs make it difficult to determine which words or actions caused an outcome, such as whether a particular code edit improved a long Claude Code session.
- Iyyer notes that RL and online policy distillation over very long trajectories remain difficult public research problems, with context summarization or truncation among proposed approaches.
- With reward 0 for an incorrect answer and 1 for a correct answer, incorrect rollouts contribute no update under the basic REINFORCE loss.
- Correct rollouts receive the ordinary negative log-likelihood update, effectively treating each successful sampled response as an SFT target.
- If a model never produces a correct rollout on difficult prompts, this setup generates samples without providing a positive learning signal.
- If all rewards are positive, basic REINFORCE raises the probability of every sampled response, though by different amounts, and does not explicitly push down poor responses.
- Subtracting a scalar baseline b gives the advantage A = R - b: above-baseline rollouts are reinforced, while below-baseline rollouts are discouraged.
- A scalar baseline can be subtracted without changing the expected policy-gradient estimate when it does not depend on the sampled action.
- Group Relative Policy Optimization, introduced with DeepSeek R1 in 2024, samples g responses for the same prompt and compares their rewards.
- GRPO computes each advantage by subtracting the group mean reward and dividing by the group reward standard deviation, producing normalized relative scores.
- For four binary-reward math rollouts with rewards 1, 0, 0, 1, mean-centering gives positive advantages to correct responses and negative advantages to incorrect ones.
- If all four responses in a GRPO group are correct, their centered advantages are zero and the group produces no update.
- If every response is incorrect with reward zero, centered advantages are also zero; prompts that are too hard or too easy waste sampling compute.
- Training benefits from prompts where the current model sometimes succeeds and sometimes fails, but model improvement can make previously useful prompts too easy.
Transcript excerpt
0:01 Okay. Let's get started everyone. So today we have a couple things to talk about. We're going to finish up our derivation of the reinforce algorithm, which is the first of these policy gradient algorithms we'll be covering for language model post training. Before we resume with where we left off in the last class, just an update that your homeworks are due later this week and the next homework will be going out shortly after. So I would expect either on Thursday or Friday. It's a fun little homework where you will actually use one of these
0:50 reinforcement learning algorithms, GRPO, to do some reinforcement learning on a tiny base model. So this is all on Google Colab. The models are super small. The task we're going to be working on is like can the model spit out a sentence with this keyword and the some like provided length, like maybe an eight word sentence mentioning this word. This is actually quite a difficult task for a tiny model and we'll see how like the hyperparameters in these reinforcement learning algorithms make a big difference in performance. So it's more intended to get you familiar with how these things actually work in
1:38 practice. Like we will talk about all the math behind these algorithms, but in practice they're very finicky and you need to try a bunch of things to actually get them to optimize properly. So that's the idea. The homework is not particularly time consuming. We provide you most of the code already. The goal is mainly for you to run of experiments and do some analysis over what this algorithm is actually doing to your your model. Okay. Great. So we left off in the middle of this kind of complicated algorithm last time. So I'll just do a quick review so that
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Mohit Iyyer.