EfficientML.ai Lecture 14 - LLM Post-Training (MIT 6.5940 Fall 2026)
Overview
Post-training turns a pretrained language model into a useful, aligned assistant through supervised fine-tuning (SFT), human preference optimization, and reinforcement learning. The lecture explains RLHF, PPO, DPO, and GRPO; parameter-efficient methods including LoRA, QLoRA, and BitDelta; and multimodal training with VILA and VILA-U, then connects prompt engineering and retrieval-augmented generation (RAG) to practical model use.
Key takeaways
- SFT provides a supervised instruction-following starting point, while RLHF and DPO use human preference comparisons to favor better responses that are difficult to capture with automatic metrics.
- GRPO compares sampled responses within a group, and its rollout phase can take over 85 to 90 percent of training time; speculative decoding can accelerate the process by over 2× without changing the target model's output distribution.
- LoRA reduces trainable parameters through low-rank weight updates that can be merged at inference, while QLoRA combines those updates with a 4-bit frozen backbone to reduce memory substantially.
- VILA's visual-language performance depends not just on adding an image encoder, but also on interleaved image-text training, adapting the language model, retaining text-only data, and using sufficient image resolution.
- VILA-U uses discrete visual tokens in an autoregressive model to support both visual understanding and generation, with residual quantization helping preserve visual detail.
- RAG provides changing or specialized information through embedding-based retrieval and optional reranking, grounding a model's answer in selected external passages without retraining.
Chapters
- Pretraining builds general capabilities and world knowledge, while post-training uses smaller, targeted datasets to teach instruction following and safety.
- The lecture covers supervised fine-tuning (SFT), reinforcement learning, parameter-efficient fine-tuning (PEFT), multimodal models, prompt engineering, and retrieval-augmented generation (RAG).
- Post-training has become increasingly important and can consume compute comparable to pretraining.
- SFT trains on prompt and desired-response examples, applying loss to response tokens rather than the prompt; examples teach helpfulness and safe refusal.
- RLHF collects human comparisons of candidate responses and trains a reward model to score preferred answers above rejected ones.
- PPO optimizes reward while using a KL penalty against a reference model; its pipeline maintains policy, reward, and reference models.
- DPO uses preferred and rejected response pairs to increase the preferred response's relative probability, avoiding a separately trained reward model and PPO stage.
- Reasoning models such as OpenAI o1 and DeepSeek R1 use reinforcement learning to improve multi-step problem solving.
- Inference-time scaling gives a model a larger token budget to work through math, coding, and other reasoning tasks.
- The raspberry spelling example illustrates how intermediate checking can catch an initially incorrect count and revise the answer.
- GRPO begins by sampling multiple responses to the same question, then scores them with task rewards.
- It calculates each response's advantage relative to the group's average reward and uses a KL term to limit drift from the reference model.
- In the example with 5551, two incorrect answers receive reward 0 and the correct answer receives reward 1, guiding future generations toward the correct solution.
- Autoregressive rollout generation can account for over 85 to 90 percent of GRPO training time, while batched inference and training are comparatively efficient.
- Variable response lengths leave GPU workers idle while they wait for the longest rollout to finish.
- The described approach trains a small speculative draft model during otherwise idle periods and uses it to accelerate target-model generation.
- The lecture reports over 2× RL training speedup while preserving the baseline reward curve through lossless speculative decoding.
- BitFit updates only bias terms, making it inexpensive and competitive on small to medium datasets, but less flexible than full fine-tuning on larger datasets.
- TinyTL keeps the main network fixed and adds a lightweight residual branch, emphasizing low activation memory for on-device training.
- Transformer adapters add trainable bottleneck modules that project hidden states down and back up while the main model remains frozen.
- Adapter bottleneck size controls capacity and storage, but adapter layers add operations to the inference path.
- LoRA represents a weight update as two low-rank matrices, reducing trainable parameters when rank r is much smaller than the original matrix dimensions.
- Initializing one matrix randomly and the other to zero preserves the pretrained model's initial behavior while allowing the LoRA branch to learn.
- At inference, the low-rank update can be merged into the original weights, avoiding the extra inference layers associated with adapters.
- Diffusion-model examples show LoRA adapting a shared model to styles such as Chinese ink and concepts such as panda-shaped sushi.
- QLoRA trains FP16 LoRA adapters against a frozen 4-bit quantized backbone, reducing memory use while retaining higher-precision trainable parameters.
- For a 7B-parameter model, the lecture compares about 14 GB in FP16 with about 3.5 GB for 4-bit weights, before other training overhead.
- QLoRA also uses double quantization and paged optimizers to reduce quantization overhead and handle GPU memory spikes.
- BitDelta represents each fine-tuned model's weight difference with one-bit signs and learned tensor scales, letting multiple task variants share a Mistral 7B base.
- VILA maps image features from a vision transformer through a projector into the language model's embedding space.
- Its training recipe progresses through projector alignment, vision-language pretraining, and high-quality supervised fine-tuning.
- Unfreezing the language model during vision-language pretraining improves few-shot visual performance compared with keeping it frozen.
- Interleaved image-text data and text-only instruction examples help multimodal in-context learning while preserving language skills.
- VILA follows visual demonstrations, such as mapping examples of animal counts to the count of cats in a new image.
- It combines image recognition with world knowledge in a logo example, associating NVIDIA with GPUs.
- A chain-of-thought example identifies beer bottles and menu prices across images before calculating the total cost.
- The lecture attributes VILA's benchmark gains to its training recipe, including interleaved image-text data, text-only SFT examples, and adequate image resolution.
- Dynamic-resolution tiling splits images into fixed-size crops and includes a global thumbnail, improving detail-sensitive tasks such as OCR.
- A separate high-resolution encoder can provide keys and values for cross-attention, supporting global-to-local interaction without hard tile boundaries but increasing training complexity.
- VILA-U represents visual inputs as discrete tokens and trains one autoregressive model for image and video understanding as well as generation.
- Residual quantization uses multiple codebooks to encode remaining visual error, helping discrete tokens retain both semantic meaning and reconstruction detail.
- Prompt engineering uses instructions and context to make one foundation model perform tasks such as translation without changing its weights.
- Few-shot examples teach formats and mappings, including invented-word definitions and binary sentiment codes.
- Balanced classification examples and consistent formatting can make few-shot prompts more reliable, especially for earlier models.
- Chain-of-thought prompting, including the instruction to think step by step, encourages intermediate reasoning on complex tasks.
- The lecture describes a role-play jailbreak anecdote and notes that safety training and system-level checks can reduce vulnerability to such attacks.
- For diffusion models, adding style, setting, lighting, and composition constraints progressively steers images toward the intended result.
- Positive prompts specify desired content, while negative prompts discourage artifacts such as deformation, haze, noise, and low contrast.
- Examples evolve from a forest house to a styled riverside scene and from a lab robot to a cartoon logo on a white background.
- RAG supplements a model's potentially stale or incomplete parametric knowledge with current information stored in an external database.
- An embedding model maps documents and queries to vectors, and a retriever finds passages with similar representations.
- An optional reranker refines passage selection so fewer, more relevant tokens reach the language model.
- The model receives the original question and selected evidence to produce a grounded answer without retraining.
- The recap connects SFT, RL, PEFT methods such as LoRA and QLoRA, multimodal models such as VILA and VILA-U, prompt engineering, and RAG.
- Final projects are completed in groups of 3 to 5 and include a proposal, poster presentation, and final report.
- Poster presentation slots are selected through a first-come, first-served signup, and the poster must be submitted before the first presentation date.
- Students can choose from more than 10 suggested project topics or propose their own through the course Canvas system.
Transcript excerpt
0:01 Okay. Uh good afternoon, everyone. Uh let's get started for today's lecture. So, today we are going to talk about uh lecture 14 on LM post-training. So, previously we already discussed how large language models are built and uh you know, deployed with some uh quantization and compression techniques. So, in this lecture we will focus on techniques makes uh uh pre-trained language model more uh powerful, useful, and better aligned to humans' preference and uh requirements. So, we will cover uh several domains such as sup- uh supervised fine-tuning, so SFT. Uh we will also talk about uh reinforcement learning and some uh parameter-efficient fine-tuning methods.
0:48 We will also talk about some uh probably uh very uh lightweight method to make large language model works better such as prompt engineering and uh retrieval arguments generation. So, let's first uh take a look at where is the post-training in the overall LM training process. So, here we have a life cycle of the large language model and we begin by uh collecting some pre-training data, very large pre-training corpus, and chance a base model using uh substantial compute. And later uh pre- a post-training happen- happens afterward with a much uh smaller, more targeted data set. Uh its purpose is to teach the model how to follow instructions, behave safely, and performs the task we care about.
1:36 And finally, uh we have some deployment organizations uh to improve the latency and memory efficiency uh as we have always been uh talking about. And pre-training creates a general capability and the world knowledge inside the large language models. Post-training turns that capability into a uh useful product or assistant at into a many applications. So, nowadays uh post-training uh typically requires much smaller data set compared to pre-training. It's already It's already become more and more
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, MIT HAN Lab.