Save this video free

SFT and LoRA

Mohit Iyyer · 1:02:45 · Watch on YouTube

Part of University of Maryland CMSC 723. See all 11 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

Mohit Iyyer surveys supervised fine-tuning data, including Tulu 3 and FLAN, and explains how synthetic responses from teacher models can transfer capabilities to smaller students. He contrasts the LIMA paper's 1,000-example superficial alignment hypothesis with evidence that SFT can add facts but risks catastrophic forgetting, then derives LoRA as a memory-efficient alternative that trains low-rank adapters instead of updating every model weight.

Key takeaways

Chapters

0:00 Lecture Plan, SFT Topics, and Homework 1
1:43 Tulu 3 Shows the Breadth of Instruction-Tuning Data
6:03 FLAN Recasts NLP Tasks as Instructions
10:41 Synthetic Math Reasoning and the Importance of Verification
15:21 Coding, Safety Examples, and Prompts for Reinforcement Learning
19:20 Downstream Tasks, In-Context Learning, and SFT
23:40 LIMA's 1,000 Examples and Superficial Alignment Hypothesis
29:02 Where Superficial Alignment Falls Short
33:43 Teacher-to-Student Distillation with Synthetic Responses
38:01 Distillation Beyond Answers: Reasoning Traces and Internal States
45:07 Why Full-Parameter SFT Is Costly
50:29 LoRA's Low-Rank Updates, Adapter Sharing, and QLoRA

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:05 All right. Uh sorry about that. So, we finally got this working. All right, let's get started um everyone. Um sorry for the delay. All right, so today I uh wanted to continue where we left off last uh time on supervised fine-tuning. We're going to talk about um like data sets that are commonly used for SFT at scale. Uh talk a bit about distillation and then focus on uh parameter efficient tuning methods. Um So, before we do anything, um oh, I should also mention uh we are done with homework one and will release it either today or tomorrow morning. Um it'll be due 2 weeks from

0:53 now. Um it's pretty straightforward uh implement uh like the uh forward uh prop phase of a transformer um and train it on a toy task. Um this will have an autograder on GradeScope. So, basically, you're graded on the quality of your implementation. If your transformer reaches like 90 or 95% um accuracy on the test set, um then you get full credit. Otherwise, you get no credit. So, you're basically just um uh it's not that scary. Uh you you will be able to um get the credit pretty easily, um but uh you have like, you know, unlimited tries to to do this. So, um don't worry about it.

1:43 Okay. Uh so, before I start with the iPad, I wanted to talk about um the SFT data mix that we see um in modern models. And so, if you remember from last time we talked about instruction tuning, right? Where the goal is train the model on as many different instructions as possible with the hope that at test time the model will be able to generalize to new instructions. So, I wanted to show you you know what some of these instructions

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Mohit Iyyer.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.