Save this video free

UUtah CS 6340 NLP | Fall 2026 | Post-training: SFT

UofU Data Science · 1:19:09 · Watch on YouTube

Part of University of Utah CS 6340. See all 19 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

Post-training adapts a pretrained language model for instruction following, conversation, capability, and alignment, with supervised fine-tuning (SFT) as the foundation and preference or verifiable-reward training as later stages. The lecture explains SFT’s next-token objective, chat formatting, data sources and limitations, then introduces RLHF reward modeling, where human comparisons teach a model to score responses without requiring people to write ideal answers.

Key takeaways

Chapters

0:00 Course Logistics and Practicum Preparation
2:01 Why Pretrained Models Need Post-Training
10:24 SFT, Preference Learning, and Verifiable Rewards
13:55 Post-Training at Scale Versus Task-Specific Fine-Tuning
17:46 SFT Uses Next-Token Prediction on Demonstrations
25:51 Chat Templates Encode Roles and Assistant-Only Loss
34:08 SFT Examples Teach Tool Use and Reasoning Traces
42:07 Human-Written and Benchmark-Derived SFT Data
49:18 Rejection Sampling and Synthetic Reasoning Data
54:36 Multilingual SFT and Participatory Data Creation
58:29 SFT Limitations Motivate Preference Training
1:02:22 Fine-Tuning Methods, LoRA, and Resource Constraints
1:07:11 RLHF Learns from Reward Scores on Model Outputs
1:11:07 Why Human Preferences Can Train a Reward Model
1:12:30 Reward Criteria Are Hard to Specify as Rules

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:00 started. Uh we are noticeably smaller number which always happens uh in November as things get uh busier. Um yeah, I hope you have a nice weekend. There is a busy month ahead of us also very technical. So I know your energy levels are probably lower than where they were when we started but hang in there. it's going to lease I hope be fun if you're interested in this topic and it's all very you know recent and relevant. So you know this these are useful stuff for uh you know when you go on a job market and and preparation for interviews and things like that. Um I sent an announcement about practicum just to remind you that it's happening on Wednesday. Um I hope you're working on your practice assignments. No one

0:48 shared any fun thing. So I'm desperate for someone to be like, I came up with this question or I found this web page. Uh so if you if you have something to share, do it for me uh for not for anyone else. Um uh the practicum itself, uh please check at the end of the practice assignment. I kind of listed a few things that you should focus on. This is just a hint. You can imagine that that's kind of what's going to be what you will need to code. Um, so make sure that coding those parts goes a little bit more uh smoothly, right? Uh, but then my goal isn't that you kind of overfit uh if I if I may use the technical term on the practice assignment. Uh, the really the key point of that current assignment is to kind of try that like full pipeline from

1:37 creating the data to trying different uh different choices. This is I think kind

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, UofU Data Science.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.