CMU AI Agents 2026: 9. Reinforcement Learning Basics
Overview
Daniel Fried connects supervised fine-tuning to reinforcement learning for agents, using a number-search task with a hidden integer from 1 to 16 and four guesses to show why agents should optimize expected task reward rather than imitate one demonstration. He derives the REINFORCE policy gradient, explains how baselines reduce gradient variance, and introduces REST-EM, GRPO, and Dr. GRPO as practical ways to learn from sampled trajectories.
Key takeaways
- Supervised fine-tuning maximizes the likelihood of demonstrated actions, while reinforcement learning directly maximizes expected task reward and can distinguish successful from failed behavior.
- REINFORCE uses the score-function identity to train through sampled discrete actions and black-box environments, weighting generated-action log-probability gradients by observed rewards.
- A baseline dependent only on interaction history can be subtracted without biasing the expected policy gradient, while reducing the variance of individual gradient samples.
- GRPO estimates a prompt-specific baseline by comparing multiple rollouts for the same task, avoiding the need for a separate value model.
- Dr. GRPO omits reward-standard-deviation scaling to retain task-difficulty information and uses a fixed token-normalization constant to avoid length-related bias.
- Binary-reward methods need successful rollouts and reward variation to learn; demonstrations, curriculum, guided prompts, and intermediate reward shaping can provide useful signals.
Chapters
- Daniel Fried introduces the first of three reinforcement-learning lectures, beginning with basic policy-gradient methods.
- In number search, the agent must guess a hidden integer from 1 to 16 within four attempts, using higher-or-lower feedback.
- A correct guess earns reward 1; using all four attempts without success earns reward 0.
- The course will apply reinforcement learning in Assignment 3 to number search and a Minecraft-like crafting environment.
- Supervised fine-tuning maximizes the log probability of demonstrated actions conditioned on their preceding history.
- Demonstrations can come from people, stronger models, or the agent's own trajectories, but each shows only one way to solve a task.
- Task mismatch arises because many action sequences can succeed, while SFT treats undemonstrated alternatives alike.
- Data mismatch and exposure bias make SFT struggle with failed examples and histories created by the model's own mistakes.
- Reinforcement learning samples the agent's own trajectories, evaluates their rewards, and updates the policy to favor higher-reward behavior.
- Optimizing expected reward directly addresses task mismatch, while reward-sensitive updates can use both successful and failed trajectories.
- On-policy training exposes the model to histories it generates itself, helping it recover from its own errors.
- Fried connects post-training reinforcement learning to improvements in language-model behaviors such as repetition loops.
- A policy πθ maps the interaction history to a distribution over the next action; for text generation, actions may be tokens or tool calls.
- A trajectory contains actions, observations, and rewards, with probabilities determined by both the agent policy and the environment.
- The environment can be treated as a black box: agents sample outcomes by acting without needing a known transition model.
- The training objective J(θ) is expected trajectory reward under the policy and environment, which can be viewed as shifting probability mass toward successful trajectories.
- Reinforced self-training, or REST-EM, alternates between sampling trajectories and fine-tuning on successful ones.
- With binary rewards, REST keeps trajectories with reward 1 and filters out trajectories with reward 0 before applying the SFT objective.
- The method reuses standard SFT machinery and resembles policy-gradient training, but stores a growing dataset rather than updating only on fresh rollouts.
- If the current policy cannot produce successful trajectories, demonstrations, easier curriculum tasks, or intermediate rewards can help bootstrap learning.
- Direct backpropagation fails because actions are sampled discrete outputs and most agent environments are non-differentiable.
- REINFORCE, introduced by Ronald Williams in 1992, applies the score-function identity to the expected-reward objective.
- The log-derivative trick rewrites the gradient as an expectation of reward multiplied by the gradient of log trajectory probability.
- The resulting expectation can be estimated from sampled trajectories instead of enumerating the enormous space of possible interactions.
- The policy-gradient estimate averages reward-weighted log-probability gradients across sampled agent trajectories.
- Environment observations and rewards do not depend on language-model parameters, so their terms disappear when differentiating trajectory probability.
- For an action expressed as multiple language-model tokens, sum the token log probabilities; exclude prompt, scaffold, and environment tokens from the policy loss.
- Standard REINFORCE typically uses one trajectory per task instance, then updates the model on the batch.
- Plain REINFORCE gives no update for a trajectory with reward 0, even when learning to avoid its actions would be useful.
- An advantage measures an action's expected outcome relative to the value expected from the current policy at that history.
- In number search, after guesses 8 and 12 with corresponding feedback, guessing 10 can be better than guessing 11 because it divides the remaining candidates more effectively.
- Baselines can be constants, running reward averages, or learned value models, and they change the reward scale used for the policy-gradient update.
- Subtracting a baseline that depends only on the current history leaves the expected policy gradient unchanged.
- The baseline term has zero expected contribution because the policy's action probabilities sum to 1.
- A valid baseline can therefore reduce variance without introducing bias into the gradient estimate.
- For tasks with intermediate rewards, reward-to-go uses only rewards from the current point onward to reduce variance.
- Fried analyzes a one-step policy with sigmoid parameter θ, where action A has reward 1 and action B has reward 0.
- At probability 0.25 for A, the exact expected gradient is 0.1875, but baseline-free samples are 0.75 for A and 0 for B.
- Using a constant baseline of 0.5 changes the sample gradients to 0.375 for A and 0.125 for B while keeping their expectation at 0.1875.
- The closer individual samples are to the true gradient, the less noisy and more stable repeated policy updates become.
- Group Relative Policy Optimization estimates expected reward without a separate value model by sampling multiple rollouts for the same task instance.
- A group typically contains about eight trajectories; each trajectory's reward is centered by the mean reward for that instance.
- The original GRPO formulation also divides by the group's reward standard deviation, though many implementations omit that scaling.
- Advantages must be computed within each task's group so the baseline reflects the policy's expected performance on that particular prompt.
- Each rollout receives an advantage based on its reward relative to other rollouts for the same task instance.
- The advantage scales the generated-token log probabilities before differentiation, extending REINFORCE to multiple samples per task.
- The lecture focuses on GRPO's advantage calculation; the original method also uses clipping and policy regularization to stabilize updates.
- A group with no reward variation provides no relative-advantage signal, even when all its trajectories succeed or all fail.
- Dr. GRPO subtracts the group mean reward without dividing by the reward standard deviation, preserving information about task difficulty.
- It replaces division by the generated sequence's token count with division by a fixed constant, such as the maximum allowed token count.
- The update assigns a centered advantage to generated tokens while normalizing by that constant.
- These changes are presented as ways to avoid biases associated with reward scaling and answer length.
- If every rollout fails or every rollout succeeds, GRPO has no within-group reward spread and therefore no relative-advantage learning signal.
- Possible remedies include choosing different task instances, adding demonstrations, guiding prompts, using reward shaping, or fine-tuning before reinforcement learning.
- REST filters successful trajectories, REINFORCE scales log-probability gradients by reward, and GRPO centers rewards against same-prompt rollouts.
- On-policy training addresses exposure bias but requires costly environment interaction and produces noisy gradients, motivating further stabilization methods.
Transcript excerpt
0:00 Okay, so let's go ahead and get started. So today is going to be about an intro to reinforcement learning which will build on the SFT for agents uh that we had last week. This is going to be the first of three lectures that we'll have on reinforcement learning. And so in this first one, we're going to be talking about basic poly gradient methods, which are ways where agents interact with the environment and get feedback and then update their weights to make things that were successful that got high reward more likely and things that got low reward less likely. We'll then have a second lecture which will actually be next week on uh more advanced RL algorithms talking about off policy methods about um some of the
0:49 things that you need to do to stabilize RL and get it to work. And then our third lecture will be on RL systems and going into some of the practical frameworks and you know things that you need to think about and pay attention to in hyperparameters when you're training these agents. And all of these should set you up uh pretty well for assignment three, which is going to be focused on training models with reinforcement learning in two settings. So the first setting is going to be a simple kind of sandboxed setting, which we'll actually use as a running example in this lecture. It's going to be called number search. And the second one is going to be like a simple kind of uh crafting environment that's a little bit like Minecraft. So for these, they're definitely going to require more compute than your previous assignments have. Um
1:36 so you'll be using the compute credits more heavily, but uh you should have the foundations to get started on those
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Graham Neubig.