Save this video free

REINFORCE (cont'd)

Mohit Iyyer · 1:08:25 · Watch on YouTube

Part of University of Maryland CMSC 723. See all 11 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

Mohit Iyyer derives REINFORCE for language-model post-training, showing how the score-function identity turns an intractable expected-reward gradient into the sample-based estimator E[R∇θ log πθ(Y|X)]. The resulting loss is negative reward times rollout log-probability; Iyyer connects it to weighted SFT, explains sparse-reward and token-level credit-assignment problems, then introduces reward baselines and GRPO’s group-wise reward normalization.

Key takeaways

Chapters

0:00 GRPO Homework: Reinforcement Learning on a Tiny Language Model
2:10 REINFORCE Setup: Prompts, Rollouts, and Scalar Rewards
8:00 Why Sampling Makes the Expected-Reward Gradient Difficult
15:00 The Likelihood-Ratio Trick Produces the REINFORCE Estimator
24:00 A Language-Model Rollout Probability Is a Product of Token Probabilities
29:00 From Expected Reward to the REINFORCE Training Loss
34:00 How REINFORCE Relates to Pretraining and Supervised Fine-Tuning
37:00 Whole-Response Rewards Create a Long-Sequence Credit-Assignment Problem
47:00 Binary Rewards Turn Correct Rollouts into SFT Examples
52:00 Reward Baselines Create Positive and Negative Advantages
58:18 GRPO Normalizes Rewards Across Rollouts for One Prompt
1:01:46 GRPO Needs Prompts with Mixed Outcomes to Produce Updates

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:01 Okay. Let's get started everyone. So today we have a couple things to talk about. We're going to finish up our derivation of the reinforce algorithm, which is the first of these policy gradient algorithms we'll be covering for language model post training. Before we resume with where we left off in the last class, just an update that your homeworks are due later this week and the next homework will be going out shortly after. So I would expect either on Thursday or Friday. It's a fun little homework where you will actually use one of these

0:50 reinforcement learning algorithms, GRPO, to do some reinforcement learning on a tiny base model. So this is all on Google Colab. The models are super small. The task we're going to be working on is like can the model spit out a sentence with this keyword and the some like provided length, like maybe an eight word sentence mentioning this word. This is actually quite a difficult task for a tiny model and we'll see how like the hyperparameters in these reinforcement learning algorithms make a big difference in performance. So it's more intended to get you familiar with how these things actually work in

1:38 practice. Like we will talk about all the math behind these algorithms, but in practice they're very finicky and you need to try a bunch of things to actually get them to optimize properly. So that's the idea. The homework is not particularly time consuming. We provide you most of the code already. The goal is mainly for you to run of experiments and do some analysis over what this algorithm is actually doing to your your model. Okay. Great. So we left off in the middle of this kind of complicated algorithm last time. So I'll just do a quick review so that

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Mohit Iyyer.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.