RLHF
Overview
Mohit Iyyer explains how Proximal Policy Optimization (PPO) reuses expensive rollouts while limiting policy drift, building on REINFORCE advantages and probability ratios. He also introduces PPO’s learned value-function baseline, its per-token predictions and memory costs, and how GRPO trades that value network for additional rollouts; human-preference reward models are deferred to the next class.
Key takeaways
- PPO's policy ratio, πθ(y|x) / πold(y|x), makes it possible to optimize the current policy with rollouts sampled from an older policy.
- With ε = 0.2, PPO's common clipping interval is 0.8 to 1.2; the clipped objective limits probability changes in the direction supported by the advantage.
- PPO can reuse a rollout batch for multiple gradient steps, commonly four or eight in the examples discussed, reducing the need for costly new decoding.
- PPO's value network predicts final reward from each token prefix, enabling per-token advantages but requiring an additional large model.
- The described PPO training setup keeps four model copies in memory: policy, value network, reward model, and frozen SFT checkpoint.
- GRPO replaces the learned value baseline with statistics from multiple rollouts, trading value-network memory and training for additional sampling compute.
Chapters
- The lecture moves from reinforcement learning for tasks such as math and coding, where rewards are verifiable, toward training with subjective human preferences.
- Mohit Iyyer previews RLHF reward models and future lessons on encouraging models to reason before giving a final answer.
- OpenAI's RLHF work is linked to ChatGPT's emergence as a capable instruction-following model.
- An advantage is the rollout reward minus a baseline; positive advantages should raise a rollout's probability, while negative advantages should lower it.
- GRPO estimates the baseline from multiple rollouts for the same prompt, using the group's mean reward and normalization by its standard deviation.
- Group-relative advantages compare each rollout with the other outputs sampled from that prompt.
- Sampling multiple completions from a large transformer is expensive because each generated sequence is decoded token by token.
- REINFORCE is on-policy: after a gradient update, the current policy differs from the policy that generated the rollouts.
- PPO addresses the cost of discarding data by allowing multiple updates while controlling how far the policy moves.
- PPO, or Proximal Policy Optimization, was introduced in 2017 for applications such as game-playing agents and robot control.
- Its central idea is to take multiple small gradient steps on the same batch of rollouts, then stop when the updated policy has moved too far.
- GRPO can be understood as a PPO variant that uses group-derived baselines rather than PPO's learned value baseline.
- PPO compares the current policy's probability of a rollout with the old policy's probability using the ratio πθ(y|x) / πold(y|x).
- The ratio weights the advantage while training on samples drawn from the old policy.
- In the expectation, the old-policy probability cancels with the denominator of the ratio, yielding an objective for the current policy using old-policy samples.
- Before any parameter update, the current policy and rollout-generating policy are identical, so the policy ratio equals 1.
- At that first update, PPO reduces to the familiar REINFORCE-style objective weighted by the advantage.
- After the update, the policies differ, and the ratio allows further optimization using the same rollouts.
- PPO clips the policy ratio to a range around 1, commonly 0.8 to 1.2 when ε is 0.2.
- The clipped surrogate objective takes the minimum of the unclipped and clipped advantage-weighted terms.
- Once a sample moves sufficiently far in the direction encouraged by its advantage, clipping makes its gradient contribution flat.
- A positive advantage means the rollout is better than the baseline, so PPO aims to increase its probability relative to the old policy.
- A negative advantage means the rollout is worse than the baseline, so PPO aims to reduce its relative probability.
- The clipping threshold limits movement in the desired direction, while the minimum operation preserves updates when the policy has moved in the wrong direction.
- A batch gradient combines competing examples, so one rollout with positive advantage can still become less probable after an update.
- PPO avoids clipping that wrong-way movement, allowing later gradient steps to correct it.
- Sequence probability decomposes into token probabilities, so implementations can calculate and clip policy ratios separately at each token position.
- The trust region is the interval from 1 − ε to 1 + ε around a policy ratio of 1.
- Clipping stops further updates after a rollout's probability has moved far enough in the direction its advantage recommends.
- RL libraries may take four or eight gradient steps on a rollout batch before sampling new data.
- Unlike GRPO's group statistics, PPO's baseline comes from a trained value function, also called a value network.
- The value function predicts the eventual rollout reward from each prefix, estimating how promising the prompt and partial response are.
- PPO can use one rollout per prompt with this learned baseline, instead of sampling a group to calculate baseline statistics.
- The value network predicts the final sequence reward at each token position, producing a different baseline for each prefix.
- Each token's advantage is the sequence-level reward minus the value prediction at that token.
- The value function is trained by regression, typically minimizing squared error between each prefix prediction and the observed final reward.
- A value network can be initialized from the SFT or other language-model checkpoint, with a scalar regression head added to its final-layer representation.
- The PPO setup described keeps four model copies: the policy, value network, reward model, and frozen SFT checkpoint for regularization.
- The value network is updated alongside the changing policy, while the reward model is generally trained offline and kept fixed.
- GRPO avoids storing a value network but spends extra decoding compute to sample multiple rollouts per prompt.
- A value function could improve credit assignment by recognizing an error early, but this depends on accurate reward predictions from partial reasoning traces.
- Mohit Iyyer cites findings associated with VinePPO that value predictions on individual tokens in math reasoning can be close to random.
- For math and coding, weak token-level value estimates help explain why many teams favor GRPO; the lecture leaves reward-model details for the next class.
Transcript excerpt
0:10 Okay, let's get started everyone. So today we are going to be um covering the PO algorithm which is uh going to introduce a couple new concepts to what we discussed when we were talking about uh reinforce um and then we're going to start getting into uh RHF which involves trained reward models that replace the verifiable reward that we've been discussing um for uh most of the RL stuff that we've covered so far. Um okay, so let's get started. Uh and kind of where we're going with this is uh so far we've looked at uh an algorithm that
0:58 can be used just as we discussed to help models improve at uh math problem solving and coding and other kinds of tasks where we have a clear clean verifiable signal. Now we're going to look at how can we use the same set of algorithms to improve tasks where we don't have such a clean uh reward anymore and we actually have to take into account like human subjective preferences uh while training our model. Um kind of where we're going after this is um once we've covered the basic learning algorithms, we're going to start focusing on uh reasoning behavior in particular. So how we can use these um algorithms that we've developed to
1:48 actually kind of modify the uh operation of the model such that it kind of thinks before it produces the final answer. So, so far we've just been covering the case where the model receives a prompt and then immediately generates the response. But as you all have uh likely experienced when you're working with modern language models, there is uh some period of time where they're thinking right where they're figuring out like how to structure the response. They might be using tools um like web search
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Mohit Iyyer.