UUtah CS 6340 NLP | Fall 2026 | Post-training: RLHF/PPO
Overview
The lecture explains RLHF as a two-stage process: train a reward model from human preference pairs, then use reinforcement learning to update an SFT language model toward higher-reward responses while limiting drift from its reference model. It derives the preference-ranking loss and policy-gradient objective, then builds PPO’s practical machinery, including a value network, per-token advantages, clipped probability ratios, and a KL penalty, explaining why PPO requires substantial compute and is difficult to reproduce.
Key takeaways
- A reward model learns from pairwise human preferences by making the preferred response’s scalar score exceed the rejected response’s score, using a negative log-sigmoid ranking loss.
- RLHF optimizes expected reward but constrains the policy with KL divergence from a frozen SFT reference, because excessive drift can erase useful capabilities and expose reward-model weaknesses.
- Policy-gradient training needs sampled responses rather than teacher-forced reference tokens, making autoregressive generation a major source of RLHF’s training cost.
- PPO stabilizes repeated updates on sampled batches with clipped probability ratios, while a value network and GAE provide per-token advantage estimates.
- PPO’s four-model setup, repeated forward and backward passes, and generation overhead help explain why it is harder to reproduce than supervised fine-tuning.
Chapters
- The remaining course topics include alignment and efficiency, with later coverage of quantization, parameter-efficient fine-tuning, caching, and attention variants.
- The lecture focuses on preference learning and RLHF, building on supervised fine-tuning as a way to make language models follow instructions.
- The stated alignment goal is to make models less likely to produce harmful actions that developers would not endorse.
- A preference example contains a prompt and at least two responses, with an annotator indicating which is preferred or whether they are tied.
- Multiple responses can be sampled from the same model using nucleus, or top-p, sampling.
- Selecting between two ChatGPT responses is an example of a user providing a preference signal.
- Preference data can come from product interfaces, hired annotators, or existing datasets such as HelpSteer.
- The lecture cites a HelpSteer dataset with about 40,000 preferences as a starting point for experimentation.
- The labor behind preference collection has raised concerns, including reports of Kenyan annotators paid $2 per hour and precarious contracts without basic benefits.
- A reward model can start from a pretrained transformer whose language-modeling head is replaced by a scalar scoring head.
- Hugging Face provides sequence-classification model classes, including `AutoModelForSequenceClassification`, with a one-output linear layer for a single reward score.
- The prompt and response are concatenated and passed together through the model to produce a score.
- For each prompt, the model scores a winning response and a losing response, aiming to assign the preferred response the higher score.
- A sigmoid applied to the score difference turns the ranking into a probability, following the Bradley-Terry preference model.
- The training loss is the negative log-sigmoid of the preferred-minus-rejected reward difference, averaged over preference pairs.
- Reward-model quality can be evaluated by the fraction of held-out pairs where the preferred response receives the higher score.
- Preferences can be subjective: annotators may disagree about which color, writing style, or open-ended answer is better.
- Verifiable math and code problems provide clearer reward signals than open-ended writing, where defining quality remains an active research challenge.
- A reward model can rank several generated candidates at inference time, a strategy called best-of-N and related to rejection sampling during training.
- In the LLM setting, the policy is the language model, the action is the next token, and the state is a prompt plus a partial or complete response.
- The vocabulary forms the action space, while a reward model scores generated responses.
- Tool-using systems may execute generated actions through a harness, but the language model still produces the tokens that specify those actions.
- The policy model, written as πRL with parameters θ, is initialized from an SFT model and updated to increase expected reward.
- A frozen copy of the SFT model serves as a reference for measuring policy divergence.
- The combined objective rewards high-scoring responses while penalizing KL divergence from the reference, helping preserve general capabilities and limit reward hacking.
- Large policy shifts can push generations outside the reward model’s training distribution, making its scores less reliable.
- The expected reward can be written as a sum over responses weighted by their probability under the current policy.
- The log-derivative trick rewrites the gradient using reward multiplied by the gradient of the response log-probability.
- Sampling responses gives a Monte Carlo estimate of the gradient, the core policy-gradient method used by REINFORCE, introduced in 1992.
- Positive rewards increase the probability of sampled responses, while negative rewards push their probability down.
- Supervised fine-tuning uses teacher forcing to maximize the probability of the reference next token in a training example.
- RLHF must generate complete responses by sampling from the current policy so they can be scored by the reward model.
- Autoregressive sampling makes training slower than ordinary supervised fine-tuning and can require more GPUs to maintain training speed.
- A baseline reduces the tendency to reinforce every response when all rewards are positive; subtracting the batch mean is a simple example.
- A learned value network estimates expected reward and provides a baseline, adding another model to the training setup.
- Advantages are computed per token as estimated return minus the value prediction, with Generalized Advantage Estimation, or GAE, used to estimate returns.
- PPO reuses sampled response batches for multiple updates, but repeated updates can make the current policy drift from the policy that generated the batch.
- The ratio compares each token’s probability under the updated policy with its probability under the previous policy.
- Clipping this ratio to a range set by ε, often around 0.2, limits how much a token’s probability can increase or decrease in one update.
- The clipped objective, often called PPO’s clip loss, applies the advantage while capping excessively large policy changes.
- PPO’s KL-divergence penalty is commonly folded into token-level rewards, with the final token also receiving the sequence-level reward from the reward model.
- The training loop samples prompts, generates completions, scores them, computes reference-model probabilities, shapes rewards, and calculates returns and advantages.
- Policy updates use the clipped PPO loss, while the value network is trained with mean squared error on returns.
- PPO can require four models in memory: a frozen reward model, a frozen reference model, a trainable policy, and a trainable value network.
- Training combines four forward passes, two backward passes, and sampled response generation, making the procedure slow and operationally complex.
- InstructGPT and Llama 2 Chat are cited as models trained with PPO; the lecture notes that public reports for many later frontier models do not clearly specify PPO.
- Hugging Face provides a PPO trainer that abstracts much of the implementation, though choosing hyperparameters and making training work remain difficult.
- The lecture recommends Professor Daniel Brown’s advanced AI course for deeper reinforcement-learning coverage.
- Nathan Lambert’s post-training course is recommended for detailed material on post-training methods, including PPO.
Transcript excerpt
0:00 Okay. Uh we can get started. How was how was your weekend? Hope it was great questions because today we have anything but an easy lecture. So I hope you are ready for it. Today we also have a professor Phillips here who will just observe what I'm doing and send a little report to the to the department. So you know you can ignore him. I hope it's fair to say. Um okay. Uh before I jump into dun dun dun reinforcement learning um I want to mention that we are not finished grading but we are about to be finished grading the last practicum and one thing that we noticed is that the core that this course look pretty good uh and I want to
0:49 kind of uh let you know that I didn't intentionally make the practicum easier in any way. So it wasn't my intention to you know boost the course or anything. So I think this is really a reflection of you kind of getting a more comfortable with them and really studying and it it shows. So great job and we are going to release scores uh really really soon and I'm going to then uh after we release this course I'm also going to share the final practice assignment for uh practicing for the final uh practicum. We still have two quizzes and then we are over with any kind of evaluation. As a reminder, there is no final exams. So once we have the final lecture, uh basically the course is done. You will still need to check your final grade and that's set up.
1:37 >> Yes, two. >> Yes, one is going to be on the alignment stuff and the other one is going to be on the efficiency. Um we still have lots
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, UofU Data Science.