Save this video free

EfficientML.ai Lecture 16 - Vision Transformer (MIT 6.5940 Fall 2026)

MIT HAN Lab · 1:18:13 · Watch on YouTube

Part of Massachusetts Institute of Technology 6.5940 EfficientML.ai. See all 19 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

Zhuoyang Zhang explains how Vision Transformers turn images into patch tokens, then surveys window, linear, and sparse attention for efficient high-resolution perception. For image generation, he covers vector-quantized tokens, parallel autoregressive decoding, and next-scale prediction, including EfficientViT-SAM’s reported 49× speedup and methods that reduce generation to far fewer dependent steps.

Key takeaways

Chapters

0:00 ViT Patch Embeddings Turn Images into Transformer Sequences
5:18 Why High-Resolution Segmentation Needs Efficient Vision Models
11:22 Swin Transformer Uses Shifted Windows for Local Attention
15:23 EfficientViT Combines Linear Attention with Local Convolutions
22:33 EfficientViT-SAM Replaces SAM’s Heavy Image Encoder
26:04 SparseViT Prunes Low-Importance Image Windows
32:28 Autoregressive Image Generation and Vector-Quantized Tokens
39:46 Why Raster-Order Image Decoding Becomes a Latency Bottleneck
42:09 Position Queries Enable Flexible Parallel Autoregressive Decoding
48:27 Training and Fused Inference for Parallel Autoregressive Models
56:14 Locality-Aware Ordering Improves Parallel Image Generation
1:04:26 VAR Predicts Image Scales Instead of Individual Tokens
1:05:40 VAR’s Residual Tokenizer Builds Coarse-to-Fine Image Codes
1:13:17 VAR Generation Predicts Successive Token Maps in Parallel
1:15:26 HART Adds Continuous Residuals to Discrete Image Tokens

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:01 All right. Let's get started. Good afternoon everyone. I'm Zhuoyang Zhang and today we are going to discuss vision transformers. As previously we mainly talk about language models today and in the next week we will talk about something about computer vision. And today's vision transformer we will talk about recognizing images and also how to generate images. And here is today's plan. First we will understand how a vision transformer turns an image into a sequence of tokens. And then we will study three approaches to efficient perception, window attention, linear attention and sparse attention. And the second half we will move to image generation. We will introduce

0:50 discrete visual tokens and also autoregressive modeling. Then we will introduce two efficient techniques including parallel decoding and next skill prediction. And as I said another critical part of vision transformer is diffusion model and we will cover this in the next week. Okay, let's start with the basic vision transformer. We usually call it a ViT. We already know that how a transformer processes a sequence. The first question is actually how to represent a two dimensional image as a sequence that the same transformer architecture can process. Here I give you a example and look at the image on the left. We divide it into into non overlapping

1:38 square patches. Then we arrange those patches in a sequence. You can think of each patch actually as a visual counterpart of a word token. The main idea is actually simple. Once the image becomes a sequence of vectors we can apply the transformer we already learn. And let's work the dimensions. The image I gave you here is 96 by 96 pixels. And each patch we make it 32 by 32. So we have three patches along each side. So the sequence contains nine patch

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, MIT HAN Lab.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.