EfficientML.ai Lecture 17 - Diffusion Model (Part I) (MIT 6.5940 Fall 2026)
Overview
The lecture develops diffusion models from their Gaussian noising and denoising process through classifier-free guidance and latent diffusion, then shows how Deep Compression Autoencoders (DCAE) and linear attention reduce generation cost. It connects these techniques to Sana image and video generation, Sana World Model, ForeAct robot planning, and LongLive streaming video, citing results such as up to 106× image-generation acceleration and LongLive 2.0 rates above 45 frames per second.
Key takeaways
- DDPM training can avoid simulating every forward step: sample a timestep and Gaussian noise, construct x_t directly using alpha_bar_t, and train the network with noise-prediction MSE.
- Classifier-free guidance removes the need for a separate classifier by combining conditional and unconditional predictions, but requires two model evaluations per denoising timestep.
- DCAE's factor-64 spatial compression yields roughly 64 times fewer latent positions than factor-8 compression, while staged low-resolution and latent-interface training helps preserve reconstruction quality.
- Sana's ReLU linear attention changes multiplication order to avoid a full N-by-N attention matrix, making long sequences more practical for high-resolution image and video generation.
- LongLive maintains streaming video continuity with anchor frames and cached history, while LongLive 2.0 combines NVFP4 and W4A4 optimizations to exceed 45 frames per second.
Chapters
- Discriminative models perform tasks such as image classification and object detection, while generative models learn data distributions to create images or videos.
- Diffusion is introduced alongside GANs, VAEs, and autoregressive generation as a method for image and video synthesis.
- The lecture roadmap moves from denoising and conditional generation to latent diffusion, DCAE, linear attention, and long-context video.
- The forward process adds Gaussian noise at each timestep, with the variance controlled by a beta schedule.
- Writing alpha_t = 1 - beta_t and multiplying the alpha values yields a direct sampling formula: x_t = sqrt(alpha_bar_t) x_0 + sqrt(1 - alpha_bar_t) epsilon.
- As t increases, alpha_bar_t decreases, weakening the original image signal until the sample approaches standard Gaussian noise.
- The reverse process is learned: a neural network predicts the noise in x_t so the sample can be stepped toward x_(t-1).
- Training samples a clean image, a random timestep, and Gaussian noise, then minimizes mean squared error between predicted and inserted noise.
- Inference repeatedly predicts and removes noise from a Gaussian sample, making pixel-space generation costly because denoising runs over many steps.
- Classifier guidance uses the gradient of a separate classifier to move denoising samples toward a requested class; a larger guidance weight improves fidelity but reduces diversity.
- Classifier-free guidance combines conditional and unconditional noise predictions from the same diffusion model, avoiding a separate classifier.
- Classifier-free guidance runs the model twice per timestep, increasing inference work while supporting conditions such as text and pixel inputs.
- A pretrained autoencoder compresses an image into a latent representation before diffusion, then decodes the final latent back into an image.
- The denoising objective remains noise-prediction MSE, but the iterative process operates on a smaller tensor rather than every high-resolution pixel.
- Typical latent representations use substantially lower spatial resolution, reducing the number of tokens processed at every diffusion step.
- Deep Compression Autoencoder (DCAE) targets a spatial compression factor of 64, compared with a factor of 8 for the Stable Diffusion VAE discussed in the lecture.
- The stronger compression produces roughly 64 times fewer spatial positions for the diffusion model to process.
- DCAE aims to preserve reconstruction quality at this aggressive compression, improving generation speed without obvious visual degradation.
- DCAE adds explicit space-to-channel and channel-to-space residual paths alongside learned convolutional sampling, making aggressive compression easier to optimize.
- Its training recipe first trains at low resolution, adapts a small latent-interface region on high-resolution data, then applies reconstruction and gain losses to the decoder head.
- The decoder-head refinement improves reported reconstruction FID from 3.82 to 0.82; DCAE 1.5 further structures latent channels so earlier channels capture coarse structure and later channels add detail.
- Sana combines a compressed autoencoder with a diffusion transformer and replaces softmax attention with ReLU feature maps.
- Reordering matrix multiplication computes key-value products before applying queries, avoiding an explicit N-by-N attention matrix when sequence length N is much larger than head dimension D.
- The reduction in attention cost is especially valuable for high-resolution images, where the token sequence grows.
- Sana uses a compact decoder-only language model to expand short prompts into detailed descriptions of objects, colors, textures, and spatial relationships.
- Visual language models generate image captions, with CLIP scores used to select or weight captions and reduce mismatches or hallucinated details.
- The combined optimizations report roughly 20× faster generation at 1K resolution and about 100× at 4K, with a reported peak of 106× on one A100.
- Sana Video compresses spatial and temporal dimensions, generates latent video in chunks, and uses recurrent state to carry context between chunks.
- Temporal convolution supplements linear attention with local frame interactions; the lecture cites about 16× lower measured latency and a reduction from 770 seconds to 30 seconds for a 5-second 720p clip on an RTX 5090.
- Sana World Model adds six-degree-of-freedom camera controls and combines recurrent linear attention with selected softmax-attention blocks; reported results include up to 38× end-to-end acceleration and 2.4× lower GPU memory on H100.
- ForeAct uses a language model to break a task into subtasks, then generates predicted images of the robot's next state.
- A vision-language-action model uses each predicted state as guidance for selecting robot actions.
- The coffee-making example shows real observations alongside predicted next states as a robot arm operates a coffee machine.
- LongLive is a frame-level autoregressive diffusion framework that accepts new prompts during generation instead of restarting the video.
- Frame-level attention sinks keep a small set of anchor frames available beyond the local attention window, helping preserve scene identity without attending to the full KV cache.
- When instructions change, KV-cache recomputation aligns cached features with the new prompt; streaming long-video tuning reduces quality drift as generation continues.
- LongLive 2.0 adds NVFP4 precision, balanced sequence parallelism, a streamlined training pipeline, and native multi-shot generation.
- Its inference stack combines W4A4 quantization, an NVFP4 KV cache, parallel dequantization, and asynchronous VAE decoding.
- The lecture reports more than 2× faster training, 1.8× faster inference, roughly half the memory of BF16, and generation above 45 frames per second.
Transcript excerpt
0:01 Uh okay, good afternoon everyone. Uh today we are going to talk about the lecture 17 uh diffusion model. Uh this is uh first part of the our lecture on uh diffusion model and on Thursday we will have the uh second half uh about the course content on the topic. So today we will have very detailed discussion of the definition of diffusion model, how is it built and some applications and optimizations regarding to this kind of powerful model for multi-modal generation, image generation and video generation. Uh so first of all let's begin with
0:50 some backgrounds of generative learning or we will talk about the basics between basically distinctions between discriminative and generative learning. So a discriminative model learns decision boundary or conditional prediction. For example we have talked about some tasks such as image classification, object detection, such kind of things and all these are [snorts] uh discriminative discriminative learning, discriminative models and generative model can be something different. It tries to learn the underlying data distribution so that we can sample from sample some new examples from the original distribution.
1:38 Specifically we can use generative model to output some for example image data, video data as shown in this example. So we have data distribution here and we train the model and we use the model to do some inference, do some sampling so we can get some examples from the original original for example image distribution, so we can get a cats here. So that is from the original distribution different kinds of cats. And here is a more detailed definition of these two kind of models, the discriminative ones and the generative ones. So as we have talked the discriminative models for tasks such
2:26 as classification and object detection etc. And the generative models ports image
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, MIT HAN Lab.