EfficientML.ai Lecture 18 - Diffusion Model (Part II) (MIT 6.5940 Fall 2026)
Overview
Diffusion inference can be accelerated along three complementary axes: sparse attention, low-bit quantization, and fewer denoising steps. The lecture covers Sparse VideoGen and its semantic-clustering extension, Radial Attention's O(L log L) attention pattern, SVDQuant and Delta-DiT for quantization, QuantVideoGen for two-bit KV-cache compression, and AnyFlow for flexible-step video diffusion distillation.
Key takeaways
- Sparse VideoGen uses attention-head patterns to preserve local spatial or temporal interactions; profiling about 1% of queries adds roughly 3% overhead, while matched sparse patterns can retain quality at about 30% attention density.
- SVG v2's semantic clustering reduces wasted block computation: the useful share rises from 28% with position-based blocks to over 86% after semantic permutation.
- Radial Attention exploits spatial and temporal attention decay with a static diagonal-band mask to target O(L log L) computation and avoids the online profiling used by dynamic sparsity methods.
- SVDQuant makes W4A4 diffusion practical by smoothing activations and isolating weight outliers in a small FP16 low-rank branch, achieving about 3.6 times smaller models and reported speedups above 10 times on RTX 4090.
- QuantVideoGen combines semantic centroid subtraction with progressive residual quantization to compress video KV caches to two bits, reaching PSNR 28.7 while reducing a 34 GB cache to about 5 GB.
- AnyFlow learns transitions between arbitrary denoising times, so one distilled model can support different sampling-step budgets instead of being restricted to a fixed four-step or two-step schedule.
Chapters
- The lecture builds on latent compression and efficient attention, focusing on three inference strategies: sparsity, quantization, and distillation.
- Video diffusion is especially expensive because each video combines spatial image tokens with tokens across many frames.
- A five-second state-of-the-art video can exceed 70,000 tokens, making dense attention a major latency bottleneck because its cost grows quadratically with sequence length.
- Video attention maps show structured diagonals and blocks rather than uniform density; nearby frames and spatially adjacent tokens tend to receive stronger attention.
- Sparse VideoGen (SVG) is training-free and assigns attention heads to spatial patterns within a frame or temporal patterns across frames.
- SVG profiles about 1% of queries to choose a sparse pattern, adding roughly 3% overhead; matched patterns can retain quality with about 30% attention density.
- Spatial heads focus within frames, while temporal heads emphasize the same image location across nearby frames.
- Temporal tokens are strided in the original layout, so SVG reorders token indices to make selected positions more contiguous for GPU execution.
- Sparse attention provides about 1.8 times speedup, with efficient kernels and lower precision raising end-to-end speedup to roughly 2.3 times.
- A layout-transform microbenchmark reports about 1.7 times faster execution than naive sparse attention.
- SVG v2 groups and reorders query and key tokens by semantic similarity, concentrating scattered important attention entries into contiguous blocks.
- Position-based blocks can spend only 28% of computation on useful entries; semantic permutation raises that fraction to over 86%.
- K-means clusters estimate attention between centroids, and a top-k rule selects key clusters to cover a target share of attention mass.
- Cached centroids make repeated clustering updates over 70 times faster; dynamic block kernels avoid padding waste and can reduce attention density by up to 2.3 times.
- Radial Attention uses a static mask based on attention-energy decay: attention generally weakens as spatial distance within a frame or temporal distance between frames increases.
- The mask assigns dense computation to the central diagonal band and progressively lower density to wider outer bands.
- With O(L) work per band and O(log L) bands, the method targets O(L log L) attention computation without online profiling.
- Radial Attention supports training-free acceleration at a model's default video length while maintaining generation quality close to full attention.
- For four-times-longer video adaptation, Radial Attention with LoRA reaches a comparable revision reward in under 200 GPU-hours, versus over 700 GPU-hours for dense attention LoRA.
- The long-video setup reduces inference latency by about 3.7 times, and its length-extension LoRA can be combined with a separate style adapter.
- NVIDIA Blackwell FP4 Tensor Cores offer up to twice the peak throughput of FP8 and four times that of formats such as FP16 or BF16.
- Using FP4 matrix hardware requires quantizing both weights and activations, making W4A4 more challenging than weight-only quantization.
- Naive W4A4 causes severe image-quality loss, and standard smoothing recovers only part of the degradation.
- SVDQuant first applies smoothing, similar to SmoothQuant, to reduce activation outliers by migrating quantization difficulty between activations and weights.
- A low-rank branch captures difficult weight components, leaving a smoother residual for four-bit quantization; the branch can remain in FP16.
- Kernel fusion reuses inputs and outputs across the four-bit and low-rank branches, reducing the branch's inference overhead by about 1.4 times.
- SVDQuant reduces model size by about 3.6 times and reports speedups exceeding 10 times on an RTX 4090; the lecture demonstrates local inference with a 12-billion-parameter FLUX.1 model.
- Video activations shift substantially across denoising steps, so a single fixed smoothing scale can help one step while worsening another.
- Delta-DiT groups nearby spatial and temporal tokens into local 3D cubes, keeps each cube's mean token at higher precision, and quantizes the remaining differences to four bits.
- Subtracting the mean reduces activation outliers; reported model-size and inference-memory reductions are about 2.9 times and 2.3 times.
- Combined with efficient attention, Delta-DiT reports 4.6 to 7.2 times acceleration; paired with few-step distillation, the lecture cites over 100 times end-to-end speedup.
- QuantVideoGen compresses autoregressive video models' 16-bit KV caches to two bits, aiming for roughly eight times smaller storage without losing long-range visual consistency.
- Semantic-aware smoothing groups related tokens and subtracts cluster centroids, reducing quantization error from extreme cache values by roughly 1,000 times.
- Progressive residual quantization refines the approximation in stages; the first two-bit stage alone reduces mean squared error by about 5.83 times.
- The method achieves near-lossless quality with about seven times less cache memory, reducing a long-video cache from roughly 34 GB to 5 GB at PSNR 28.7; its runtime overhead stays below 4%.
- AnyFlow targets flexible inference budgets rather than a student trained for one fixed number of denoising steps, allowing users to trade speed for quality.
- Unlike consistency distillation, which maps a noisy state directly to the endpoint, AnyFlow learns transitions from time t to an arbitrary earlier time r.
- Stage one trains a LoRA student on teacher flow-map transitions, using Jacobian-vector products and shortcut transitions to make training practical.
- Stage two uses on-policy distillation: the student generates its own trajectories, then aligns its predictions with the teacher on states it will encounter at inference.
- AnyFlow supports bidirectional and causal video diffusion across models from 1.3 billion to 14 billion parameters, including text-to-video and image-to-video settings.
- Additional sampling steps can improve quality while increasing cost, and distilled models remain adaptable to robotics or driving data.
- The lecture's three main strategies are sparse attention, low-bit quantization, and distillation to reduce denoising steps.
Transcript excerpt
0:02 Uh okay let's get started for today's lecture and today we are going to talk about the uh second part of the diffusion model. So we talked about the uh basic uh definition of diffusion modeling uh in the last lecture and about about the uh how is the diffusion model the noise from the uh random noise to uh for example image or video uh set as a output. And we also talk about some uh optimization techniques such as uh the deep compression autoenccoder for uh higher compression ratio when you convert the image or video into the latent space so that you can save a lot of tokens uh during the image and video generation. And we also talk about the
0:49 uh hybrid linear attention architecture which makes uh attention operation more efficient uh on the uh multimodel generation. So today we are mainly going to talk about uh some inference optimizations uh for particularly for the diffusion modeling um we'll talk about sparity uh quantization and distillation respectively. So um first of all uh we are going to discuss some uh sparsity techniques in uh division models and the central question here is uh how to preserve the general quality while spending less computation. So you can uh easily map this kind of idea to the uh concept of
1:39 uh sparsity such as sparse attention. We have talked in previous uh lectures about uh for example long context language models and uh so observation in diffusion models are quite similar to what we met in the long context language models. We can see that uh video diffusion models can be very expensive uh as a tension layer. That's because um in video generation we have uh uh each frame is a image and a video is consist of uh a lot of
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, MIT HAN Lab.