EfficientML.ai Lecture 16 - Vision Transformer (MIT 6.5940 Fall 2026)
Overview
Zhuoyang Zhang explains how Vision Transformers turn images into patch tokens, then surveys window, linear, and sparse attention for efficient high-resolution perception. For image generation, he covers vector-quantized tokens, parallel autoregressive decoding, and next-scale prediction, including EfficientViT-SAM’s reported 49× speedup and methods that reduce generation to far fewer dependent steps.
Key takeaways
- At fixed image resolution, halving ViT patch width and height creates four times as many tokens and sixteen times as many attention pairs, making patch size a direct accuracy-efficiency tradeoff.
- Swin Transformer’s shifted windows enable cross-window information flow over depth without returning to full global attention at every layer.
- EfficientViT makes attention scale linearly with token count by replacing softmax similarity with a reorderable linear form, then uses local convolutions to recover visual detail.
- EfficientViT-SAM preserves SAM’s prompting interface and reports roughly 49× speedup at matched mask performance, including a throughput increase from 11 to 182 images per second.
- Parallel autoregressive decoding works best when each generated token is near existing context but far from other tokens generated simultaneously, balancing strong conditioning with reduced within-step dependence.
- VAR’s coarse-to-fine scale prediction and HART’s continuous residual refinement illustrate two ways to reduce sequential generation while retaining image structure and detail.
Chapters
- A 96×96 RGB image split into 32×32 patches produces 9 visual tokens, each initially a flattened 3,072-value vector.
- A shared learned projection maps each patch vector from 3,072 dimensions to a 768-dimensional embedding.
- Positional embeddings identify patch locations, and bidirectional attention lets every image patch interact with every other patch.
- Smaller patches preserve finer detail but increase token count; halving patch width and height creates 4× as many tokens and 16× as many attention pairs.
- Medical-image segmentation and autonomous-driving scenes require fine spatial detail to localize small structures such as tissue boundaries and pedestrians.
- Downsampling can erase details before the network processes them, while dense attention computation grows quadratically with token count.
- Segment Anything Model (SAM) combines an image encoder, a prompt encoder for points, boxes, masks, or text, and a mask decoder.
- SAM can return multiple candidate masks with quality scores when a prompt, such as a click on a person's shirt, is ambiguous.
- Swin Transformer divides feature maps into fixed-size windows and computes attention independently within each window, reducing cost compared with global attention.
- Hierarchical stages downsample feature maps so early levels retain local detail and coarser levels represent larger image regions.
- Fixed window boundaries restrict communication between neighboring windows, especially when an object crosses a boundary.
- Alternating regular and shifted windows lets tokens interact across former window boundaries while keeping each attention operation local.
- Standard softmax attention forms an N×N query-key matrix, which becomes costly as image-token count N grows.
- EfficientViT replaces softmax similarity with a ReLU-based linear similarity, allowing matrix multiplication to be reordered into a smaller feature-dimension matrix.
- With feature dimension fixed, the reordered calculation scales linearly with N, but the simpler attention can lose the concentrated local selectivity of softmax.
- Lightweight local convolutions add nearby visual information and help recover accuracy while linear attention supplies global context.
- EfficientViT-SAM replaces the original SAM image encoder while retaining its prompt encoder and lightweight mask decoder.
- Training first distills features from the original SAM encoder, then fine-tunes end-to-end on the SA-1B dataset.
- The reported comparison reaches roughly 49× speedup at matched performance on zero-shot COCO mask average precision.
- A throughput example rises from 11 images per second for original SAM to 182 images per second with the efficient model.
- SparseViT allocates computation by image content, keeping informative high-resolution regions while pruning less useful regions.
- Window activation pruning ranks windows using the inexpensive L2 magnitude of their activations and gathers the selected windows for computation.
- Window-level pruning creates more regular workloads than dropping arbitrary individual tokens, which helps practical execution.
- Sparsity-aware adaptation trains across different layer-wise pruning patterns, and evolutionary search selects configurations under a latency budget.
- Unlike a discriminative ViT that maps visual tokens to a label, an autoregressive generator predicts image tokens conditioned on a class label or text prompt.
- The image-token joint distribution is factorized into conditional distributions, commonly ordered in raster sequence from left to right and top to bottom.
- Vector quantization maps continuous encoder features to the nearest learned codebook vector, representing each feature as a discrete integer token.
- A codebook’s vocabulary size determines how many alternatives the autoregressive model must consider at each prediction.
- A 256×256 image represented on a 16×16 latent grid requires 256 sequential token predictions in raster order.
- Higher-resolution images require even more dependent prediction steps, making one-token-at-a-time decoding slow.
- KV caching avoids recomputing previous attention states but does not remove the dependency between successive samples.
- The lecture motivates parallel prediction as a way to shift decoding away from memory-bound serial steps toward more compute-efficient work.
- The proposed flexible method separates context tokens from generation: generated image tokens provide context, while learned position-query embeddings request tokens at target positions.
- A decoding step can request an arbitrary subset, such as positions 3 and 5, instead of always predicting the next raster position.
- The approach supports different generation orders and multiple target tokens per step while preserving conditioning on already generated image tokens.
- Separating position queries from image-token context lets the model parallelize generation without treating every target position as an ordinary next-token input.
- Training uses teacher forcing with ground-truth image tokens interleaved with position-query tokens.
- Context attention lets each query use earlier ground-truth tokens, while query attention lets simultaneously generated positions exchange information for more coherent outputs.
- A combined attention mask prevents query tokens from being treated as generated context and preserves the autoregressive constraint.
- Inference conceptually separates encoding generated tokens from decoding requested positions, but a specialized mask can fuse both operations into one forward pass.
- Attention visualizations show that raster-order image models strongly favor nearby tokens, including neighbors to the left and tokens in the row above.
- Parallelizing adjacent raster positions risks generating mutually dependent tokens independently, while random order weakens conditioning on previously generated tokens.
- Locality-aware ordering generates tokens near existing context but keeps tokens generated in the same round far apart.
- The method reports 13× faster generation while preserving ImageNet FID, 21× speedup at higher resolution, and up to 60× speedup for 1K images by reducing roughly 4,000 steps to 64.
- Visual Autoregressive Modeling (VAR) changes the prediction unit from one token to an entire spatial scale.
- Generation proceeds from a coarse image representation to finer detail, allowing multiple tokens to be predicted in parallel at each scale.
- The next-scale strategy is designed to generate large images in roughly 10 to 15 scale steps rather than a long raster sequence.
- A multi-scale tokenizer is required to represent image content as discrete token maps at progressively finer resolutions.
- At each scale, the tokenizer resizes the current feature representation, quantizes it with a shared codebook, then subtracts the reconstructed contribution to obtain a residual.
- Each finer token map encodes information left out by earlier, coarser maps, and reconstruction sums contributions from all scales.
- In the lecture’s 2×2 toy example, the 1×1 coarse scale produces token ID 4, while the 2×2 residual scale produces four additional token IDs.
- The larger VAR setup uses a 16×16 latent feature map, a 4,096-entry codebook, and up to 10 progressively larger scales.
- VAR starts from a condition, generates a coarse token map, and uses its upsampled representation as input for the next scale.
- A specialized attention mask lets tokens within one scale attend to each other and lets later scales attend to earlier scales, but not the reverse.
- Parallel prediction within each scale reduces image generation to a small number of steps and makes the process less serial and memory-bound.
- The lecture notes that good image quality still depends on tokenizer reconstruction quality, since discrete quantization can lose visual detail.
- HART introduces a hybrid tokenizer and autoregressive transformer to address detail lost through vector quantization.
- Its design treats discrete tokens as a representation of the main image structure and continuous residual features as corrections that restore finer detail.
- The tokenizer’s decoder receives quantized features on some training passes and bypasses quantization to receive continuous features on others.
- Training the decoder on both representations is intended to combine continuous reconstruction fidelity with tolerance for imperfect features generated from discrete tokens.
Transcript excerpt
0:01 All right. Let's get started. Good afternoon everyone. I'm Zhuoyang Zhang and today we are going to discuss vision transformers. As previously we mainly talk about language models today and in the next week we will talk about something about computer vision. And today's vision transformer we will talk about recognizing images and also how to generate images. And here is today's plan. First we will understand how a vision transformer turns an image into a sequence of tokens. And then we will study three approaches to efficient perception, window attention, linear attention and sparse attention. And the second half we will move to image generation. We will introduce
0:50 discrete visual tokens and also autoregressive modeling. Then we will introduce two efficient techniques including parallel decoding and next skill prediction. And as I said another critical part of vision transformer is diffusion model and we will cover this in the next week. Okay, let's start with the basic vision transformer. We usually call it a ViT. We already know that how a transformer processes a sequence. The first question is actually how to represent a two dimensional image as a sequence that the same transformer architecture can process. Here I give you a example and look at the image on the left. We divide it into into non overlapping
1:38 square patches. Then we arrange those patches in a sequence. You can think of each patch actually as a visual counterpart of a word token. The main idea is actually simple. Once the image becomes a sequence of vectors we can apply the transformer we already learn. And let's work the dimensions. The image I gave you here is 96 by 96 pixels. And each patch we make it 32 by 32. So we have three patches along each side. So the sequence contains nine patch
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, MIT HAN Lab.