EfficientML.ai Lecture 19 - Lec19-Distributed-Training (Part I) (MIT 6.5940 Fall 2026)
Overview
Distributed training makes large-model research practical by dividing work across GPUs through data, pipeline, tensor, and sequence parallelism, then coordinating those workers with communication primitives such as all-reduce. The lecture explains parameter-server bottlenecks, ring and logarithmic all-reduce strategies, and DeepSpeed ZeRO/FSDP memory sharding, which reduces replicated training state enough to raise an illustrative 80 GB GPU capacity from about 5 billion to 320 billion parameters before activation memory.
Key takeaways
- A 175-billion-parameter model's estimated 3 million GPU-hours would take about 355 years on one GPU, making multi-GPU training essential for practical research iteration.
- A parameter server centralizes weight distribution and gradient aggregation, so its O(N) worker traffic becomes a scaling bottleneck; ring all-reduce spreads communication across neighboring workers.
- For the lecture's Adam memory estimate, weights and gradients use 2 bytes each while optimizer state uses 12 bytes per parameter, making optimizer-state sharding a high-impact first step.
- ZeRO Stage 1 shards optimizer state, Stage 2 also shards gradients, and Stage 3 shards weights; FSDP implements fully sharded data parallelism in PyTorch.
- Pipeline microbatches reduce idle gaps between layer stages: the example improves GPU utilization from 25% to 57% by splitting a batch of 16 into batches of 4.
- Sequence parallelism can exceed the head-count limit of all-to-all repartitioning through ring attention, which circulates token blocks and computes cross-block attention progressively.
Chapters
- Model sizes have grown from tens of millions of parameters, such as ResNet-50, to open models exceeding two trillion parameters.
- A 175-billion-parameter model is estimated to require 3 million GPU-hours, equivalent to roughly 355 years on one GPU.
- Summit at Oak Ridge National Laboratory demonstrated a 211x speedup for video understanding training, reducing a roughly 49-hour single-node run to about 14 minutes with similar accuracy.
- Data parallelism replicates the model and splits examples across GPUs, so each replica computes gradients from different data.
- Pipeline parallelism assigns different model layers to different GPUs, while tensor parallelism partitions layer weights or operations across GPUs.
- Sequence parallelism divides a long token sequence across GPUs, addressing workloads that can exceed 100,000 tokens.
- Workers keep identical model weights but process separate data shards, producing different local gradients.
- In the parameter-server design, workers pull weights, compute gradients, push them to a central server, and receive updated weights.
- Synchronous training waits for all workers' gradients before the server aggregates them and updates the shared model.
- Scatter distributes different data pieces from one process, whereas broadcast sends the same tensor to every process.
- Gather collects tensors without combining them; reduce combines them, for example summing values 1, 2, 3, and 4 into 10.
- All-gather makes gathered data available across workers, while all-reduce combines worker values and distributes the reduced result to all workers.
- A central parameter server must send weights to and receive gradients from every worker, creating an O(N) traffic bottleneck at the server.
- Ring all-reduce communicates with neighboring workers, reducing per-worker communication demand while taking O(N) steps around the ring.
- Recursive-doubling exchanges at offsets 1, 2, and 4 spread data across eight workers in three stages, giving a logarithmic number of communication stages.
- For FP16 parameters with Adam, the lecture counts 2 bytes for weights, 2 for gradients, and 12 for optimizer state, or 16 bytes per parameter before activations.
- Naive data parallelism therefore fits about 5 billion parameters in 80 GB; ZeRO Stage 1 shards optimizer state, and Stage 2 shards optimizer state plus gradients.
- ZeRO Stage 3 shards weights as well, corresponding to PyTorch Fully Sharded Data Parallel (FSDP); the lecture's simplified estimate reaches about 320 billion parameters per 80 GB GPU across 64 GPUs.
- Assigning successive layers to separate GPUs creates pipeline bubbles when later layers must wait for earlier forward or backward passes.
- The naive four-stage example reaches only 25% utilization because GPUs remain idle while waiting for dependent computations.
- Splitting a batch of 16 into microbatches of 4 lets stages begin work sooner and raises the example's utilization to 57%.
- Tensor parallelism partitions weight matrices across GPUs, allowing each GPU to compute part of a matrix multiplication on a replicated input.
- For an MLP, the first matrix is split by columns and the next by rows, so intermediate results stay local and a reduction combines outputs at the end.
- Megatron-LM-style attention partitioning splits Q, K, and V heads across workers; projection and attention work proceeds independently, with communication at the final output reduction.
- Splitting a long sequence across GPUs distributes tokens, but full attention requires each token block to interact with other blocks.
- One approach repartitions data across attention heads using all-to-all communication; its parallelism is limited by the number of heads, such as 32 heads limiting the degree to 32.
- Ring attention rotates sequence blocks among GPUs and computes additional attention interactions at each step, avoiding a single all-to-all repartition while covering the full attention map.
- The lecture distinguishes data and sequence parallelism, which split inputs, from pipeline parallelism by layer and tensor parallelism by weight or operation.
- DeepSpeed ZeRO stages shard optimizer state, then gradients, then weights; ring all-reduce supports distributed gradient aggregation.
- Suggested references include the ZeRO paper, Ring Attention, DeepSpeed, and Megatron-LM; the next lecture is announced as a continuation of large-scale distributed training.
Transcript excerpt
0:01 Okay, good afternoon everyone. Let's get started. So I'm back. We'll jump into a new chapter about efficient training. Um let's quickly review what we have covered so far. We have covered about efficient inference from those general purpose techniques like pruning, quantization, distillation, um neural architecture search. And then we cover those um efficient AI particularly for generative models, large language models, diffusion models, latest techniques, autoregressive models, how to efficiently serve them, how to efficiently post train them. So now let's switch to a new new chapter about distributed training. How do we train a large model using a lot of GPUs? So this is the plan for today's lecture. We'll start with the background and
0:48 motivation and then uh how to paralyze? What are the dimensions we can parallel parallel? Um we'll talk about data parallelism and then communication primitives are reduced. And how do we reduce the memory in data parallelism introducing the zero and FSDP methods? We also cover pipeline parallelism, tensor parallelism, and sequence parallelism. So these are the majority the major techniques used these days to train large language models. So let's start with background and motivation. So models are getting bigger and bigger. To achieve good accuracy, you need a very big model. And the model size is growing very fast
1:36 um from the past decade. Um the model size grew from like 50 million parameter, 25 million parameter for ResNet 50 all the way to um trillions of parameters these days for like these open source models is already over two trillion parameters. And those large models take a long time to train. For ResNet 50, which has only 26 million parameters, takes about 31 GPU hours to train. But it's 175 billion parameter model that take 3 million GPU hours to train.
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, MIT HAN Lab.