Save this video free

CMU AI Agents 2026: 9. Reinforcement Learning Basics

Graham Neubig · 1:17:46 · Watch on YouTube

Part of Carnegie Mellon University 11-768. See all 14 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

Daniel Fried connects supervised fine-tuning to reinforcement learning for agents, using a number-search task with a hidden integer from 1 to 16 and four guesses to show why agents should optimize expected task reward rather than imitate one demonstration. He derives the REINFORCE policy gradient, explains how baselines reduce gradient variance, and introduces REST-EM, GRPO, and Dr. GRPO as practical ways to learn from sampled trajectories.

Key takeaways

Chapters

0:00 Course Roadmap and the 16-Number Search Task
5:00 SFT Optimizes Demonstrations, Not Task Success
12:00 On-Policy Learning Addresses SFT's Three Agent Problems
20:00 Trajectory Probabilities and Expected Reward
27:00 REST-EM: Filtering Successful Rollouts for SFT
32:00 REINFORCE Derives a Gradient Without Backpropagating Through the Environment
38:00 Implementing REINFORCE with Generated Token Log Probabilities
44:00 Advantages Compare Actions with Expected Policy Performance
52:00 History-Dependent Baselines Preserve an Unbiased Gradient
56:00 A Two-Action Bandit Shows How Baselines Reduce Variance
1:04:00 GRPO Estimates a Baseline from Groups of Rollouts
1:08:00 GRPO Adds Group-Based Advantages to Policy Updates
1:11:00 Dr. GRPO Removes Standard-Deviation and Length Biases
1:14:00 Sparse Rewards, Bootstrapping, and Training Recap

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:00 Okay, so let's go ahead and get started. So today is going to be about an intro to reinforcement learning which will build on the SFT for agents uh that we had last week. This is going to be the first of three lectures that we'll have on reinforcement learning. And so in this first one, we're going to be talking about basic poly gradient methods, which are ways where agents interact with the environment and get feedback and then update their weights to make things that were successful that got high reward more likely and things that got low reward less likely. We'll then have a second lecture which will actually be next week on uh more advanced RL algorithms talking about off policy methods about um some of the

0:49 things that you need to do to stabilize RL and get it to work. And then our third lecture will be on RL systems and going into some of the practical frameworks and you know things that you need to think about and pay attention to in hyperparameters when you're training these agents. And all of these should set you up uh pretty well for assignment three, which is going to be focused on training models with reinforcement learning in two settings. So the first setting is going to be a simple kind of sandboxed setting, which we'll actually use as a running example in this lecture. It's going to be called number search. And the second one is going to be like a simple kind of uh crafting environment that's a little bit like Minecraft. So for these, they're definitely going to require more compute than your previous assignments have. Um

1:36 so you'll be using the compute credits more heavily, but uh you should have the foundations to get started on those

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Graham Neubig.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.