Save this video free

RLHF

Mohit Iyyer · 1:12:05 · Watch on YouTube

Part of University of Maryland CMSC 723. See all 11 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

Mohit Iyyer explains how Proximal Policy Optimization (PPO) reuses expensive rollouts while limiting policy drift, building on REINFORCE advantages and probability ratios. He also introduces PPO’s learned value-function baseline, its per-token predictions and memory costs, and how GRPO trades that value network for additional rollouts; human-preference reward models are deferred to the next class.

Key takeaways

Chapters

0:00 From Verifiable Rewards to RLHF and Human Preferences
3:36 REINFORCE Advantages and the GRPO Group Baseline
10:21 Why On-Policy REINFORCE Discards Rollouts After One Update
17:08 PPO's Goal: Reuse Rollouts Without Excessive Policy Drift
22:52 The PPO Policy Ratio Corrects for Old-Policy Rollouts
29:53 Why PPO Starts with the REINFORCE Update
34:00 PPO Clipping Defines a Trust Region for Policy Ratios
40:19 How Advantage Sign Determines PPO's Update Direction
43:08 Batch Gradients and Per-Token PPO Clipping
47:28 PPO's Trust Region Enables Multiple Updates per Rollout Batch
53:36 PPO Uses a Learned Value Function as Its Baseline
57:06 Training the Value Network for Per-Token Advantages
1:02:06 PPO's Four Model Copies and GRPO's Compute Tradeoff
1:07:28 Value-Function Limits and the Shift Toward GRPO

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:10 Okay, let's get started everyone. So today we are going to be um covering the PO algorithm which is uh going to introduce a couple new concepts to what we discussed when we were talking about uh reinforce um and then we're going to start getting into uh RHF which involves trained reward models that replace the verifiable reward that we've been discussing um for uh most of the RL stuff that we've covered so far. Um okay, so let's get started. Uh and kind of where we're going with this is uh so far we've looked at uh an algorithm that

0:58 can be used just as we discussed to help models improve at uh math problem solving and coding and other kinds of tasks where we have a clear clean verifiable signal. Now we're going to look at how can we use the same set of algorithms to improve tasks where we don't have such a clean uh reward anymore and we actually have to take into account like human subjective preferences uh while training our model. Um kind of where we're going after this is um once we've covered the basic learning algorithms, we're going to start focusing on uh reasoning behavior in particular. So how we can use these um algorithms that we've developed to

1:48 actually kind of modify the uh operation of the model such that it kind of thinks before it produces the final answer. So, so far we've just been covering the case where the model receives a prompt and then immediately generates the response. But as you all have uh likely experienced when you're working with modern language models, there is uh some period of time where they're thinking right where they're figuring out like how to structure the response. They might be using tools um like web search

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Mohit Iyyer.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.