UUtah CS 6340 NLP | Fall 2026 | Post-training: DPO, GRPO, RLVR, reasoning
Overview
The lecture explains how post-training moved from PPO-based RLHF to DPO and back toward reinforcement learning, especially GRPO with verifiable rewards. It connects these methods to reasoning models such as OpenAI o1 and DeepSeek R1, showing how test-time compute, tool use, and agent harnesses extend LLM capabilities while introducing challenges such as reward hacking and poor writing habits.
Key takeaways
- DPO removes the separately trained reward model by substituting the optimal policy into the Bradley-Terry preference objective, where the intractable partition-function terms cancel.
- GRPO replaces PPO's learned value-network baseline with relative rewards across multiple responses to the same prompt, reducing memory needs while preserving clipped policy updates.
- RLVR uses executable or formal checks such as unit tests, answer matching, and Lean verification, making it effective for coding and math but poorly suited to nuanced goals such as good writing.
- Reasoning models remain autoregressive next-token predictors; longer thinking traces can improve performance, and models can learn to allocate more inference tokens to harder prompts.
- Tool-using agents depend on a harness that executes token-generated actions, returns observations, and manages context; the LLM itself does not execute the tools.
- Reward optimization can create unintended behavior, including convoluted writing, overstated certainty, and reward hacking, so reward design and monitoring remain central post-training challenges.
Chapters
- Post-training follows pretraining and supervised fine-tuning (SFT), using preference or reward signals to shape model responses.
- PPO-based RLHF can require a policy model, value network, reward model, and reference model, making training difficult to reproduce.
- DPO reformulates the RLHF objective to train directly from offline preference pairs without a separately trained reward model.
- The RLHF objective combines expected reward with a KL-divergence constraint that keeps the policy near its reference model.
- Substituting the optimal policy into the Bradley-Terry preference model makes the intractable partition-function terms cancel.
- The resulting DPO loss uses the policy and reference models' likelihoods for preferred and rejected responses, then minimizes negative log preference probability.
- DPO became a widely used post-training recipe for open-weight models around 2024, following pretraining and SFT.
- Nathan Lambert's reported experience with OLMo and Tulu 3 characterizes DPO as useful when engineering resources are limited and as effective for relatively smaller models.
- DPO can produce a solid model, but it is not necessarily the route to the strongest model; the lecture notes that current frontier recipes are not publicly known.
- SFT raises the likelihood of responses already represented in its training examples, potentially limiting exploration of different solution strategies.
- DPO also learns from a fixed offline preference dataset, so its updates reflect preferences collected before training.
- Online RL can sample new responses and reinforce strategies that earn high rewards, although the limits of generalization from SFT remain an open research question.
- Group Relative Policy Optimization (GRPO) samples multiple responses for one prompt and computes each response's advantage relative to the group's mean reward.
- The commonly described advantage subtracts the group mean and divides by the group reward standard deviation, while retaining PPO-style ratio clipping.
- Using group statistics instead of a learned value network removes one model from the training setup; standard-deviation normalization can introduce optimization bias.
- Verifiable rewards replace a separately trained reward model with checks such as code unit tests, known math answers, or Lean proof verification.
- These checks reduce memory requirements and can eliminate the need for human preference data in the task being optimized.
- GRPO and reinforcement learning from verifiable rewards (RLVR) are distinct: GRPO is an optimization algorithm, while RLVR describes the reward source.
- A reward optimized for a narrow task can encourage awkward prose, including hidden agents, vague phrases, and misplaced emphasis.
- The lecture contrasts direct wording such as “the duck asked for grapes” with convoluted rewrites that obscure who did what.
- Post-training can shift epistemic markers from uncertainty words toward certainty words, potentially affecting whether people verify model outputs.
- PPO typically uses a learned value network for its baseline and computes token-level advantages; GRPO uses a group of responses and a response-level advantage copied across tokens.
- GRPO requires multiple samples per prompt, often around 16 in the examples discussed, and normalizes advantages by group reward standard deviation.
- Both approaches use clipped policy ratios; the lecture describes PPO implementations as folding the KL penalty into reward, while GRPO commonly includes a separate KL term.
- With verifiable rewards, GRPO can reduce the setup to a trainable policy and a reference model, without separate value or reward models.
- Binary rewards can discard distinctions between partially correct and fully correct answers; graded signals such as the fraction of passed unit tests can retain more information.
- A single response-level score does not identify which reasoning steps helped, motivating process reward models that assign signals to intermediate steps or tokens.
- Whether frontier models still use KL-divergence constraints is unclear; open-weight reports continue to include them, while some researchers explore reducing or removing the penalty.
- Reasoning models still generate one token at a time, but they produce a thinking trace before the final answer.
- Thinking markers such as a special think token are vocabulary tokens added by the tokenizer; the trace is trained as part of the model's output.
- A displayed reasoning summary from services such as ChatGPT or Claude is not necessarily the model's full internal trace, and generated traces may not faithfully reflect computation.
- Prompting an untrained model to explain its reasoning may work inconsistently; reasoning SFT instead supplies examples containing an instruction, a trace, and a solution.
- RL can reinforce traces associated with verified correct solutions, encouraging the policy to generate more successful reasoning patterns.
- The lecture presents SFT followed by RL as a common outline, while noting that practical training pipelines combine methods in more complex ways.
- Test-time compute measures inference resources used after training, including how many tokens a model generates before answering.
- OpenAI's o1 release materials showed performance improving as reasoning-token budgets increased on evaluated problems.
- Pass@1 measures the fraction of problems solved by one sampled response; pass@k generalizes the metric to multiple sampled attempts.
- DeepSeek R1 helped demonstrate that extended autoregressive reasoning can work without relying on classical search methods such as Monte Carlo tree search.
- Models can learn that harder prompts warrant more reasoning tokens, making trace length a dynamic decision rather than a fixed output length.
- Providers retain system-level controls, including limiting reasoning-token budgets or suppressing the thinking-token path.
- Tool-using models generate tool calls as tokens, but the surrounding program or harness executes the requested action and returns the tool output to the model.
- An agent trajectory can alternate between reasoning, action, and observation, such as a tool call followed by a response and more reasoning.
- SFT can train on successful tool-use trajectories, while RL can sample multiple trajectories in a sandbox and reward those that pass task verification.
- A harness handles tool schemas, action execution, observations, and context management; protocols such as the Model Context Protocol aim to standardize tool interfaces.
- Cross-session memory can personalize explanations, for example by remembering that a user is a CS major and avoiding explanations of basic programming concepts.
- Multiple agents can collaborate on difficult tasks, but long reasoning traces and large agent groups can make problem solving expensive.
- Reasoning traces can help monitor reward hacking, such as a model proposing to alter an evaluation script, though traces may omit or contradict actual behavior.
- A Microsoft AI report describes specialist models trained for STEM, agentic tasks, and helpfulness and safety, then combined into a broader system.
- STEM and agentic training can use verifiable rewards, while helpfulness and safety rely on a learned reward model for nuanced judgments.
- Specialist training and model blending illustrate why current pipelines are more varied than a simple SFT-to-RLVR sequence.
- RL exploration comes from sampling responses beyond a fixed preference dataset and reinforcing those that score well, but reward exploitation remains difficult to characterize.
- GRPO's group-mean baseline distinguishes responses with above-average rewards, even when rewards are binary, by assigning different relative advantages within a sampled group.
- Textual feedback could convey richer guidance than a zero-or-one reward, but computing gradients through discrete text is difficult and the approach remains an open research area.
Transcript excerpt
0:01 Okay, we can get started. That was a cold bike ride to get here. We had like hot summer and cold winter in one semester. Uh, how are you all doing? Good. Cool. Um, okay. I don't have any special announcements. Do you have any questions for me in terms of logistics? Okay. Yeah. The only announcements I said this morning was that our quiz is happening uh on Monday and then I made the screenshots of lectures that will be covered in the quiz which are all uh about uh post training basically. Yeah. Okay. Then let's continue. Uh this is the structure that I keep bringing back. Our kind of whole goal of postraining is to go over algorithms and methods for uh how to tell how to teach models uh what
0:50 you want. And we started with supervised fine-tuning. Last time we learned about PO uh which uses human feedback in the form of human preferences to change the model to give us uh responses that have a higher reward. And uh today we are going to continue with two more uh methods. Uh they are obviously going to be shorter because last time I need a full lecture to cover PO. Um and it's the whole course is coming kind of coming together at this point. I would say our goal was to learn what large language models are and when we finish post training basically know all the major development stages. Uh but there are just few more things uh I I want to mention that will complete uh your knowledge about what the LLMs are. Um and then we are going to jump with the
1:40 next three lectures into how to make them you know usable uh given the amount of users we have and given the length of
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, UofU Data Science.