UUtah CS 6340 NLP | Fall 2026 | Post-training: SFT
Overview
Post-training adapts a pretrained language model for instruction following, conversation, capability, and alignment, with supervised fine-tuning (SFT) as the foundation and preference or verifiable-reward training as later stages. The lecture explains SFT’s next-token objective, chat formatting, data sources and limitations, then introduces RLHF reward modeling, where human comparisons teach a model to score responses without requiring people to write ideal answers.
Key takeaways
- SFT changes pretrained model weights using instruction-response demonstrations but retains the same next-token negative log-likelihood objective used in language modeling.
- Chat templates turn role-labeled messages into model-specific special-token strings, and SFT typically applies loss only to assistant responses, not system or user messages.
- Tool-use behavior can be taught through SFT examples that include tool definitions, assistant tool calls, tool observations, and final answers.
- Synthetic demonstrations, distillation, and rejection sampling make SFT data generation more scalable, while diverse teachers and human data can help preserve less common behaviors.
- RLHF reward models learn scalar response scores from human comparisons, avoiding the need for people to author a perfect reference response for every prompt.
- SFT’s teacher-forcing distribution shift and its penalty on alternative valid phrasings help explain why later post-training stages learn from preferences or verification signals.
Chapters
- The practicum is scheduled for Wednesday, and students are encouraged to work through the practice assignments.
- The practice assignment emphasizes coding selected pipeline components and understanding the full process from data creation to testing different choices.
- Pretraining uses next-token prediction on large-scale internet text, which usually does not pair a user instruction with a desired response.
- Post-training changes model weights after pretraining to support general instructions, requested styles, and multi-turn conversation.
- The lecture frames post-training around usability, capability, and alignment, with coding and math increasingly central to capability.
- Verifiable outcomes, such as compiler results or Lean proof checks, provide useful training signals for coding and mathematics.
- Supervised fine-tuning is the consistent first post-training stage, teaching instruction following and conversational behavior.
- Preference learning compares multiple sampled responses and uses human choices to favor preferred outputs, including safer responses.
- More recent training can use deterministic or model-based evaluators, such as compilers and Lean, to check whether outputs are correct.
- The lecture distinguishes preference-based methods, including RLHF and DPO, from reinforcement learning with verifiable rewards.
- Post-training is still fine-tuning, but it is performed at larger scale with richer instruction and conversation data than typical 2018 to 2022 task-specific fine-tuning.
- A post-trained model can be fine-tuned again for a specific application if its data and evaluation set justify doing so.
- Further fine-tuning generally requires an open-weight model; a prompted frontier model may perform better on reasoning tasks.
- Private deployment, API costs, latency, and simpler task requirements can make a smaller fine-tuned model useful.
- SFT starts from a pretrained base model and trains it on instruction or prompt pairs with high-quality target responses.
- The training objective remains negative log-likelihood: maximize the probability of each target response token given the prompt and preceding tokens.
- In PyTorch, logits and softmax probabilities are produced in a batched forward pass, then indexed by the target vocabulary IDs to compute loss.
- The key change from pretraining is the supervised instruction-response data, not a new language-modeling objective; the lecture contrasts up to 100,000 SFT examples in 2023 with nearly a million in some current settings.
- A conversation is represented as a list of messages with roles such as system, user, and assistant, each paired with content.
- A tokenizer’s `apply_chat_template` function converts those messages into a single string with model-specific special tokens.
- Role markers and end-of-turn markers teach message order and stopping behavior, while loss is calculated on assistant tokens rather than system or user text.
- Hugging Face chat-template utilities avoid manually reproducing each model’s token format, which differs across models such as Mistral.
- Agent-oriented SFT data can include tool descriptions, assistant tool calls, tool responses, and a final answer.
- In the weather example, the model should call a weather tool for current conditions rather than rely on outdated parametric knowledge.
- Reasoning text visible in a conversation may not faithfully reflect internal computation; omissions or contradictions are known as chain-of-thought unfaithfulness.
- Monitoring systems inspect reasoning traces, observations, and tool calls, but their reliability depends partly on whether models report relevant reasoning honestly.
- Early instruction data came from human annotators writing prompts and high-quality responses, an approach associated with labor and compensation controversies.
- NLP benchmarks can be reformatted as instructions; FLAN is an example of a large instruction-following collection built from existing tasks.
- Synthetic data is generated by programs or models, making it faster and cheaper to produce than human-written responses.
- A stronger model can generate demonstrations for a smaller model through distillation, though model providers’ terms of service may restrict using their outputs for competing models.
- Rejection sampling generates multiple responses to one prompt, scores them with a judge, and keeps a high-scoring response for training.
- At inference time, sampling multiple candidates and selecting the best is commonly called best-of-n.
- Reasoning data can be generated by prompting strong models on math, coding, or science problems, then filtering outputs; non-reasoning examples help preserve quick responses to simple questions.
- Model-collapse concerns focus on losing rare patterns in synthetic data; mixing human examples, varied teachers, and diverse quality filters can help retain breadth.
- Multilingual instruction data can come from translated NLP benchmarks, translated English SFT examples, or prompts and responses created by native speakers.
- Machine-translated examples can sound unnatural and may miss cultural references or values that matter to local users.
- The lecture describes native-speaker involvement as preferable for culturally relevant data and points to Cohere’s multilingual resources.
- Participatory research involves language communities and local researchers in data construction and modeling; Masakhane is cited for African-language work.
- Teacher forcing trains on correct preceding tokens, but at inference the model conditions on its own generated tokens, creating a distribution shift.
- Next-token loss penalizes every alternative to the demonstrated token, including valid phrasings that differ from the single reference response.
- In 2022 and 2023, weaker models and the difficulty of writing expert responses limited synthetic SFT data.
- Preference judgments are easier for people to provide than full demonstrations, motivating methods that score a model’s own outputs.
- A post-class discussion contrasts updating all pretrained weights with parameter-efficient approaches such as LoRA.
- Low-rank adapters reduce the number of trainable parameters, while quantization can reduce memory demands on limited GPU hardware.
- The discussion emphasizes that full-model training and fine-tuning choices depend on available compute and the model’s scale.
- Unlike SFT demonstrations, reinforcement learning can explore generated responses, so training must manage how far updates move from the starting checkpoint.
- RLHF became important when strong synthetic teachers were unavailable and collecting preference labels was easier than commissioning expert-written responses.
- A reward model assigns a scalar score to a response generated by the current language model; reinforcement learning uses that score to update the model.
- The RLHF response is not trained by maximizing likelihood against a human reference answer; it is evaluated and improved through the reward signal.
- Reward modeling and policy updating are separate problems: first learn response scores, then learn how to change model weights using them.
- A reward model aims to map a prompt and response to a scalar value that reflects response quality.
- Human comparisons provide training signal: a person identifies which of two responses is preferable without having to specify every reason.
- The next step is to learn scores that rank preferred responses higher than alternatives across diverse prompts and outputs.
- Possible response criteria include avoiding hate speech, hallucinations, sexism, and discrimination while favoring concise, grammatical, factual explanations.
- Preferences can conflict: some users value a strong persona, while others prefer a neutral style.
- Writing a complete deterministic ruleset and combining all criteria into one score is difficult, so preference data can teach a model these criteria implicitly.
- The lecture ends before deriving the preference-training objective, setting up the next class on how comparisons produce scalar reward scores.
Transcript excerpt
0:00 started. Uh we are noticeably smaller number which always happens uh in November as things get uh busier. Um yeah, I hope you have a nice weekend. There is a busy month ahead of us also very technical. So I know your energy levels are probably lower than where they were when we started but hang in there. it's going to lease I hope be fun if you're interested in this topic and it's all very you know recent and relevant. So you know this these are useful stuff for uh you know when you go on a job market and and preparation for interviews and things like that. Um I sent an announcement about practicum just to remind you that it's happening on Wednesday. Um I hope you're working on your practice assignments. No one
0:48 shared any fun thing. So I'm desperate for someone to be like, I came up with this question or I found this web page. Uh so if you if you have something to share, do it for me uh for not for anyone else. Um uh the practicum itself, uh please check at the end of the practice assignment. I kind of listed a few things that you should focus on. This is just a hint. You can imagine that that's kind of what's going to be what you will need to code. Um, so make sure that coding those parts goes a little bit more uh smoothly, right? Uh, but then my goal isn't that you kind of overfit uh if I if I may use the technical term on the practice assignment. Uh, the really the key point of that current assignment is to kind of try that like full pipeline from
1:37 creating the data to trying different uh different choices. This is I think kind
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, UofU Data Science.