Save this video free

CMU AI Agents 2026: 8. Supervised Fine Tuning

Graham Neubig · 1:04:59 · Watch on YouTube

Part of Carnegie Mellon University 11-768. See all 14 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

Yueqi Song presents supervised fine-tuning (SFT) as a cold-start stage for agents: it teaches general tool-use behavior and long-task execution from recorded trajectories, while reinforcement learning (RL) can specialize behavior using rewards from live rollouts. The lecture details assistant-token loss masking, trajectory and teacher selection, data diversity and filtering, the Agent Data Protocol, training pitfalls such as template drift, and why evaluation must test deployed agents rather than rely on training loss alone.

Key takeaways

Chapters

0:00 Agent Trajectories and Updating Model Weights
5:00 SFT as the Cold Start Before Agent RL
8:30 Serializing Tool Interactions and Masking SFT Loss
16:50 End Tokens, Reasoning Masks, and Token-Level Balance
22:25 Choosing Teachers and Filtering Agent Trajectories
31:15 Task Sources, Teacher Quality, and Data Scaling
37:30 Agent Data Protocol and Multi-Source Training
42:45 SFT Configurations, Expert Routing, and Sequence Packing
48:30 Why Training Loss Cannot Replace Agent Evaluation
52:50 Preparing SFT Checkpoints for Domain-Specific RL
54:50 Teacher Limits, Self-Distillation, and Recovery Examples
1:02:50 SFT Objective: Cross-Entropy with Teacher Forcing

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:00 Hi everyone. I'm really glad to uh give the lecture today. Uh it's on super wise fine tuning for agents. So before I start, I want to ask like how many of you have trained agents before? Like please raise your hand. Okay. Um then how many of you have done SFT on agents? Okay, that's actually more than I expected. Um okay so I will be go over these topics today like uh because this is the first lecture on training of agents. So I will first give a overall introduction of like what is training of agents mean and uh and then go over SFT specific topics and also like transitioning into RL which will be the next three lectures. Okay. Um this is a sample trajectory

0:53 sampled from sui gym. We can see there are different rows in the trajectory. Um so there are a row of system, row of user, row of assistant and observation. Like among these uh we can see there's only three assistant messages in this trajectory. Um so in the overall setting the agent receives an issue which is like given by the user and a couple of tools in a sandbox like for example in this case this is bash and a file edit tool. So in the process the agent produces ss and generates commands and uh reads the

1:41 observations that those commands return at eventually the repository's own tasks like unit test will determine whether the run succeeded or not. There are several methods of updating the agent that we already discussed about like first context window. Um there are different pros and cons for different methods like for context window it has high fidelity to what already happened but it is costly noisy and does not decide what matters and for

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Graham Neubig.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.