CMU AI Agents 2026: 8. Supervised Fine Tuning
Overview
Yueqi Song presents supervised fine-tuning (SFT) as a cold-start stage for agents: it teaches general tool-use behavior and long-task execution from recorded trajectories, while reinforcement learning (RL) can specialize behavior using rewards from live rollouts. The lecture details assistant-token loss masking, trajectory and teacher selection, data diversity and filtering, the Agent Data Protocol, training pitfalls such as template drift, and why evaluation must test deployed agents rather than rely on training loss alone.
Key takeaways
- Treat agent SFT as a cold start: it can teach tool-call formats and long-horizon execution, while RL can improve behavior using rewards from the agent's own environment rollouts.
- Assistant-only loss masking is a useful default because user messages and tool observations are context, but the value of training on observations or reasoning blocks should be tested for the target task.
- Select trajectories for more than final success: intermediate actions are copied by SFT, so filtering, turn-count checks, and inspection for repeated or wasteful behavior matter.
- Data diversity can matter more than repeated sampling: adding new task sources can continue improving results after extra rollouts from a fixed distribution begin to saturate.
- A high-performing teacher is not automatically a good distillation teacher; evaluate student gains because differences in reasoning style or action sequences can make demonstrations hard to imitate.
- Training loss is not a substitute for agent evaluation: test the fine-tuned model in its deployment harness with verifiable outcomes, repository-level splits, and multiple harnesses.
Chapters
- A SWE-Gym-style trajectory records system and user messages, assistant actions, and tool observations; repository tests determine whether the task succeeded.
- Updating model weights can change behavior across calls without adding prompt context, but training is slower and the resulting behavior is opaque and model-specific.
- Recorded trajectories can teach tool-call syntax, how to continue through long interactions, and how to react to tool results, including undesirable habits present in the data.
- The post-training pipeline typically moves from SFT to RL, with SFT establishing baseline agent capabilities and RL learning from model-generated rollouts and rewards.
- DeepSeek-R1-Zero applied RL directly to a base model, but DeepSeek-R1 added thousands of SFT examples after R1-Zero produced hard-to-read, mixed-language responses.
- SFT learns from token-level targets in recorded trajectories and is useful for tool formats and long tasks; RL uses trajectory-level rewards and requires a live environment and checkable outcomes.
- A chat template converts a multi-turn trajectory, including tool calls and a final answer, into one token sequence with model-specific special tokens.
- Standard agent SFT applies cross-entropy loss to assistant-generated tokens while masking user messages and environment observations as context.
- The template tracks assistant spans so the trainer receives a token sequence and a one-bit-per-token loss mask rather than the original message structure.
- Training on observations can be useful for learning execution semantics or world models, but its effect on general agent capabilities remains an empirical question.
- SFT masks must include the assistant's stop and end-of-turn tokens, whose exact representation varies across model families and chat templates.
- Reasoning blocks may be retained as context or trained as assistant output; very long reasoning can dominate shorter tool-call and execution targets.
- Balanced SFT approaches seek to preserve learning from both reasoning and tool use, while Microsoft MAI's data mixture illustrates why balancing example counts does not necessarily balance token counts.
- The lecture identifies careful agent-specific comparisons of masking choices, including training on observations, as an open research area.
- SFT data may come from a stronger teacher or the student itself; keeping all teacher outputs is plain distillation, while retaining verifier-passing runs filters for successful trajectories.
- Expert iteration uses the student to generate trajectories and keeps the runs that pass a verifier, creating a repeated self-improvement loop.
- SWE-Gym reports higher task resolution after distilling trajectories from stronger models and retaining runs that pass repository tests.
- A successful final test does not guarantee every intermediate action is worth imitating: repeated test loops, unnecessary edits, or leftover print statements can teach poor habits.
- OpenHands research found that a simple filter dropping trajectories shorter than five turns could help on average, although it can discard clean short fixes and retain longer, messy runs.
- The strongest benchmark agent is not necessarily the best teacher: GPT-5.3-Codex produced less useful student gains than some weaker-ranked alternatives in the reported comparison.
- Task sources have benchmark-specific effects; SWE-smith contributed strongly to SWE-bench Verified but less to Terminal-Bench.
- Repeating rollouts from the same task distribution can saturate, while adding new tasks through synthetic augmentation can keep performance improving; the lecture cites mixing roughly four to eight sources as a useful empirical finding.
- The Agent Data Protocol addresses incompatible dataset formats, such as web observations represented as HTML in one dataset and accessibility trees in another.
- A dataset can be converted once into the shared protocol and then into a target harness format, reducing the need for custom pairwise converters.
- The shared representation can retain both HTML and accessibility-tree observations so downstream training can choose its preferred format.
- Controlled experiments found that mixing diverse data sources improved performance across agent benchmarks compared with training on a single source, including on a source's own domain benchmark.
- Published training configurations vary widely: examples discussed include 294K-token sequences for Nemotron 3 Ultra, 128K for Microsoft MAI self-distillation, and 512K for Kimi K2 Horizon.
- Mixture-of-experts models can route most tokens to a small subset of experts when trained on narrow agent data; Microsoft MAI describes balancing routing during SFT.
- Packing whole conversations avoids truncating or splitting trajectories; examples are assigned to packs with suitable remaining capacity, and identical problems are kept apart before shuffling.
- The serving chat template must match the training template closely, even in whitespace, because template drift can alter prefixes and break tool-loop behavior.
- Teacher forcing trains each recorded next action under the trajectory's fixed context, but deployment actions change what the agent observes next.
- A model can encounter states absent from its training trajectories, so a low SFT loss does not establish that it can recover from novel errors.
- Before scaling training, verify that the pipeline can overfit about 20 examples; failure to reduce loss often points to an upstream data or masking bug.
- Evaluate in the actual harness with checkable task outcomes, split by repository when testing generalization, and test multiple harnesses because trajectories also teach system-specific conventions.
- Song frames SFT as the stage for broad agent capabilities such as tool use, while RL is better suited to domain-specific gains and improved reasoning.
- An SFT checkpoint should be selected with later RL in mind rather than optimized only for its standalone score.
- DeepSeek-R1-Zero shows that omitting SFT can leave a base model with weak usable agent behavior, while SFT alone still falls short of the combined SFT-plus-RL result.
- When no stronger teacher is available, Microsoft MAI's self-distillation is one option; otherwise, the teacher's capabilities limit what SFT can transfer.
- The lecture contrasts teacher-based SFT with RL, which learns from the model's own rollouts rather than depending on a stronger model's demonstrations.
- Intermediate wrong edits can be included when the trajectory teaches recovery and ends in a correct result, but training on mistakes can also reinforce bad habits.
- For resource-constrained runs, Yueqi Song recommends testing data choices empirically on a smaller scale; low-rank adaptation can reduce training cost, while well-resourced labs often prefer full fine-tuning.
- SFT uses the same cross-entropy objective as language-model pretraining, applied to chat-formatted trajectories and usually only to selected assistant tokens.
- Teacher forcing supplies the recorded context at each step rather than sampling the model's own actions during training.
- The distinction between recorded-context imitation and self-generated rollouts motivates the following lectures on RL and other training methods.
Transcript excerpt
0:00 Hi everyone. I'm really glad to uh give the lecture today. Uh it's on super wise fine tuning for agents. So before I start, I want to ask like how many of you have trained agents before? Like please raise your hand. Okay. Um then how many of you have done SFT on agents? Okay, that's actually more than I expected. Um okay so I will be go over these topics today like uh because this is the first lecture on training of agents. So I will first give a overall introduction of like what is training of agents mean and uh and then go over SFT specific topics and also like transitioning into RL which will be the next three lectures. Okay. Um this is a sample trajectory
0:53 sampled from sui gym. We can see there are different rows in the trajectory. Um so there are a row of system, row of user, row of assistant and observation. Like among these uh we can see there's only three assistant messages in this trajectory. Um so in the overall setting the agent receives an issue which is like given by the user and a couple of tools in a sandbox like for example in this case this is bash and a file edit tool. So in the process the agent produces ss and generates commands and uh reads the
1:41 observations that those commands return at eventually the repository's own tasks like unit test will determine whether the run succeeded or not. There are several methods of updating the agent that we already discussed about like first context window. Um there are different pros and cons for different methods like for context window it has high fidelity to what already happened but it is costly noisy and does not decide what matters and for
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Graham Neubig.