Save this video free

UUtah CS 6340 NLP | Fall 2026 | Post-training: RLHF/PPO

UofU Data Science · 1:18:59 · Watch on YouTube

Part of University of Utah CS 6340. See all 19 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

The lecture explains RLHF as a two-stage process: train a reward model from human preference pairs, then use reinforcement learning to update an SFT language model toward higher-reward responses while limiting drift from its reference model. It derives the preference-ranking loss and policy-gradient objective, then builds PPO’s practical machinery, including a value network, per-token advantages, clipped probability ratios, and a KL penalty, explaining why PPO requires substantial compute and is difficult to reproduce.

Key takeaways

Chapters

0:00 Course Roadmap and the Post-Training Focus
3:22 Preference Data: Comparing Sampled Responses
8:55 Preference Collection and Annotation Labor
11:00 Building a Scalar Reward Model from an LLM
13:29 Training the Reward Model with Pairwise Preferences
23:29 Preference Ambiguity and Reward-Model Uses
25:19 Mapping Reinforcement Learning Terms to LLMs
29:26 RLHF Objectives and the Reference SFT Model
42:38 Deriving the REINFORCE Policy Gradient
50:49 Why RLHF Cannot Use Teacher Forcing
59:20 Baselines, Value Networks, and Token Advantages
1:04:42 PPO’s Clipped Probability-Ratio Objective
1:10:17 PPO Reward Shaping and the Full Training Loop
1:14:29 PPO’s Four-Model Compute Burden and Adoption
1:17:03 PPO Tooling and Further Study

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:00 Okay. Uh we can get started. How was how was your weekend? Hope it was great questions because today we have anything but an easy lecture. So I hope you are ready for it. Today we also have a professor Phillips here who will just observe what I'm doing and send a little report to the to the department. So you know you can ignore him. I hope it's fair to say. Um okay. Uh before I jump into dun dun dun reinforcement learning um I want to mention that we are not finished grading but we are about to be finished grading the last practicum and one thing that we noticed is that the core that this course look pretty good uh and I want to

0:49 kind of uh let you know that I didn't intentionally make the practicum easier in any way. So it wasn't my intention to you know boost the course or anything. So I think this is really a reflection of you kind of getting a more comfortable with them and really studying and it it shows. So great job and we are going to release scores uh really really soon and I'm going to then uh after we release this course I'm also going to share the final practice assignment for uh practicing for the final uh practicum. We still have two quizzes and then we are over with any kind of evaluation. As a reminder, there is no final exams. So once we have the final lecture, uh basically the course is done. You will still need to check your final grade and that's set up.

1:37 >> Yes, two. >> Yes, one is going to be on the alignment stuff and the other one is going to be on the efficiency. Um we still have lots

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, UofU Data Science.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.