Save this video free

EfficientML.ai Lecture 19 - Lec19-Distributed-Training (Part I) (MIT 6.5940 Fall 2026)

MIT HAN Lab · 52:40 · Watch on YouTube

Part of Massachusetts Institute of Technology 6.5940 EfficientML.ai. See all 19 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

Distributed training makes large-model research practical by dividing work across GPUs through data, pipeline, tensor, and sequence parallelism, then coordinating those workers with communication primitives such as all-reduce. The lecture explains parameter-server bottlenecks, ring and logarithmic all-reduce strategies, and DeepSpeed ZeRO/FSDP memory sharding, which reduces replicated training state enough to raise an illustrative 80 GB GPU capacity from about 5 billion to 320 billion parameters before activation memory.

Key takeaways

Chapters

0:00 Why Distributed Training Is Necessary for Modern Models
5:05 Four Ways to Distribute Training Work Across GPUs
10:08 Data Parallelism and the Parameter-Server Training Loop
13:47 Scatter, Gather, Reduce, Broadcast, and All-Reduce
18:37 Ring and Logarithmic All-Reduce Trade Communication for Speed
26:12 DeepSpeed ZeRO and FSDP Shard Training Memory
34:22 Pipeline Parallelism Uses Microbatches to Reduce GPU Idle Time
37:31 Tensor Parallelism Splits MLP and Attention Computation
42:48 Sequence Parallelism: All-to-All Partitioning and Ring Attention
50:00 Distributed-Training Recap and Further Reading

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:01 Okay, good afternoon everyone. Let's get started. So I'm back. We'll jump into a new chapter about efficient training. Um let's quickly review what we have covered so far. We have covered about efficient inference from those general purpose techniques like pruning, quantization, distillation, um neural architecture search. And then we cover those um efficient AI particularly for generative models, large language models, diffusion models, latest techniques, autoregressive models, how to efficiently serve them, how to efficiently post train them. So now let's switch to a new new chapter about distributed training. How do we train a large model using a lot of GPUs? So this is the plan for today's lecture. We'll start with the background and

0:48 motivation and then uh how to paralyze? What are the dimensions we can parallel parallel? Um we'll talk about data parallelism and then communication primitives are reduced. And how do we reduce the memory in data parallelism introducing the zero and FSDP methods? We also cover pipeline parallelism, tensor parallelism, and sequence parallelism. So these are the majority the major techniques used these days to train large language models. So let's start with background and motivation. So models are getting bigger and bigger. To achieve good accuracy, you need a very big model. And the model size is growing very fast

1:36 um from the past decade. Um the model size grew from like 50 million parameter, 25 million parameter for ResNet 50 all the way to um trillions of parameters these days for like these open source models is already over two trillion parameters. And those large models take a long time to train. For ResNet 50, which has only 26 million parameters, takes about 31 GPU hours to train. But it's 175 billion parameter model that take 3 million GPU hours to train.

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, MIT HAN Lab.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.