Save this video free

UUtah CS 6340 NLP | Fall 2026 | Post-training: DPO, GRPO, RLVR, reasoning

UofU Data Science · 1:18:12 · Watch on YouTube

Part of University of Utah CS 6340. See all 19 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

The lecture explains how post-training moved from PPO-based RLHF to DPO and back toward reinforcement learning, especially GRPO with verifiable rewards. It connects these methods to reasoning models such as OpenAI o1 and DeepSeek R1, showing how test-time compute, tool use, and agent harnesses extend LLM capabilities while introducing challenges such as reward hacking and poor writing habits.

Key takeaways

Chapters

0:00 Post-Training Goals and the Move from PPO to DPO
3:06 DPO Derivation: Removing the Reward Model and Partition Function
11:24 DPO Adoption in Open-Weight Models and Its Limits
13:22 Why RL Can Go Beyond Fixed Preference Data
16:08 GRPO Replaces the Learned Value Baseline with Group Statistics
21:48 Verifiable Rewards for Code, Math, and Lean Proofs
25:29 How RL Rewards Can Distort Writing and Communicated Uncertainty
31:36 GRPO and PPO Compared: Models, Advantages, and KL Terms
38:23 Reward Granularity, Process Rewards, and the KL Debate
43:21 Reasoning Traces as Autoregressive Thinking Tokens
47:46 Teaching Reasoning with SFT and Reinforcement Learning
50:03 Test-Time Compute and Pass@k Evaluation
53:16 DeepSeek R1 and Learned Adaptive Reasoning Effort
57:44 Tool-Calling Trajectories and the Agent Harness
1:04:18 Agent Memory, Multi-Agent Workflows, and Safety Monitoring
1:07:21 Specialist GRPO Models and Blending Different Reward Types
1:11:09 Open Questions: Exploration, Reward Hacking, and Text Feedback

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:01 Okay, we can get started. That was a cold bike ride to get here. We had like hot summer and cold winter in one semester. Uh, how are you all doing? Good. Cool. Um, okay. I don't have any special announcements. Do you have any questions for me in terms of logistics? Okay. Yeah. The only announcements I said this morning was that our quiz is happening uh on Monday and then I made the screenshots of lectures that will be covered in the quiz which are all uh about uh post training basically. Yeah. Okay. Then let's continue. Uh this is the structure that I keep bringing back. Our kind of whole goal of postraining is to go over algorithms and methods for uh how to tell how to teach models uh what

0:50 you want. And we started with supervised fine-tuning. Last time we learned about PO uh which uses human feedback in the form of human preferences to change the model to give us uh responses that have a higher reward. And uh today we are going to continue with two more uh methods. Uh they are obviously going to be shorter because last time I need a full lecture to cover PO. Um and it's the whole course is coming kind of coming together at this point. I would say our goal was to learn what large language models are and when we finish post training basically know all the major development stages. Uh but there are just few more things uh I I want to mention that will complete uh your knowledge about what the LLMs are. Um and then we are going to jump with the

1:40 next three lectures into how to make them you know usable uh given the amount of users we have and given the length of

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, UofU Data Science.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.