EfficientML.ai Lecture 9 - Knowledge-Distillation (MIT 6.5940 Fall 2026)

Overview

Knowledge distillation transfers a stronger model’s predictive behavior or internal representations to a student, helping compact networks retain accuracy without adding inference cost. The lecture compares logit, weight, activation, gradient, sparsity, and relational matching; covers self- and online distillation for vision and language tasks; and introduces network augmentation, which temporarily expands an underfitting tiny model during training before returning it to its original size.

Key takeaways

Temperature-scaled softmax distillation transfers more than the teacher’s top class: increasing temperature flattens probabilities and reveals relative confidence across alternative classes.
When teacher and student tensor shapes differ, projection layers can align intermediate weights or features for training, but they can be discarded before inference.
Distillation is not limited to logits: activation maps, gradient-derived attention, ReLU sparsity patterns, and pairwise example relationships can all provide student supervision.
Self-distillation and online mutual learning avoid requiring a separately pretrained fixed teacher; Deep Mutual Learning lets peer networks teach one another and can improve both.
Large-model augmentation often combats overfitting, whereas tiny models are commonly capacity-limited and underfit; temporarily expanding their training network can help without increasing deployed model size.
Network augmentation and knowledge distillation provide complementary training signals, and experiments in the lecture show augmentation can improve compact-model transfer and object-detection efficiency.

Chapters

0:00 Why Tiny Models Need Knowledge Distillation

Cloud GPUs can run large ResNets and transformers, but microcontrollers may have only hundreds of kilobytes of on-device memory.
A MobileNetV2 example reaches roughly 50% training accuracy and 47% validation accuracy, compared with ResNet-50 at about 80% and 76%, illustrating the small model’s underfitting challenge.
Knowledge distillation aims to improve a compact student using a stronger teacher without requiring the teacher at inference.

4:00 Teacher–Student Logits and Temperature-Softened Targets

Teacher and student receive the same input; training combines the student’s task loss with a distillation loss from the teacher.
For a cat image, both models may predict cat, but example logits produce different confidence levels: the teacher gives about 98% cat probability while the student gives about 70%.
Softmax temperature divides logits before normalization: higher temperature flattens class probabilities and exposes more information about alternative classes.
Matching teacher and student distributions can use cross-entropy, KL divergence, or an L2 loss on logits.

9:00 Logit Distillation and Intermediate Weight Matching

Output-logit matching is the simplest approach: compare teacher and student predictions for the same input alongside the ordinary classification objective.
Direct weight matching is difficult when a large, deep teacher and a narrow student have different tensor dimensions.
A trainable projection layer can map student weights or channels into the teacher’s shape so an L2 loss can compare them.
The projection is a training aid, not part of inference; retaining the task loss prevents the student from merely copying the teacher’s weight distribution.

14:00 Feature, Gradient, Sparsity, and Relational Distillation

Activation matching compares intermediate teacher and student feature maps to encourage similar representations for the same input.
Gradient-based attention maps identify input regions that influence the prediction; a face example highlights eyes and nose, and the student can be trained to focus on similar regions.
Sparsity matching aligns activation zero patterns, such as those created when ReLU sets negative values to zero.
Relational distillation matches relationships within residual stages or pairwise distances among examples, preserving hidden-space structure even when networks differ in depth or width.

22:00 Born-Again Networks, Deep Mutual Learning, and Self-Distillation

Born-Again Networks train successive generations of the same architecture, using each generation as the next generation’s teacher.
Deep Mutual Learning trains two networks together: each alternates between student and teacher roles, and the architectures need not be identical.
On CIFAR-10 and CIFAR-100, mutual learning improves both networks compared with independent training.
Be Your Own Teacher adds intermediate classification heads and distills from the final head, improving early predictions that can support early exits and lower inference latency.

29:00 Object Detection Distillation for Features and Bounding Boxes

Detection distillation covers intermediate features, class predictions, and object localization rather than only image-level labels.
A 1×1 convolution aligns teacher and student feature-channel dimensions before feature maps are compared with L2 loss.
Foreground locations receive greater distillation weight than background regions, which are numerous and could otherwise dominate the loss.
Bounding-box guidance can be gated by whether the teacher is better than the student relative to ground truth; coordinate distributions can also be discretized and matched with classification-style losses.

35:00 Semantic Segmentation and GAN Compression

Semantic segmentation predicts a class for every pixel, so distillation can match pixel-level activations and predictions.
An adversarial discriminator can distinguish teacher outputs from student outputs; the student learns to make its outputs harder to distinguish.
GAN Compression uses a smaller generator, channel-matching projections, reconstruction loss, and adversarial loss to approach a larger generator’s output quality.
A CycleGAN example increases generation speed from roughly 12 frames per second to more than three times that rate while keeping image-editing quality close to the original.

41:00 Transformer Attention and Large-Language-Model Distillation

Transformer students can imitate teacher token-to-token attention distributions as well as hidden features and final outputs.
LLM compression can combine pruning and distillation; the lecture cites shrinking a model from 15 billion parameters to 8 billion while aiming to preserve benchmark accuracy.
Distillation targets can include MLP layers, attention maps, final language-model heads, and logits.
Teacher-generated or human-collected data can support distillation during pretraining or post-training, including training with fewer tokens.

45:00 Why Standard Data Augmentation Can Hurt Tiny Networks

Common regularization methods include Cutout, MixUp, AutoAugment, Dropout, SpatialDropout, and DropBlock.
MixUp combines images and labels; for example, a cat target of (1, 0) and dog target of (0, 1) become a mixed target of (0.5, 0.5).
Augmentation often helps large models such as ResNet-50 by reducing overfitting, but extra perturbations can make learning harder for capacity-limited tiny models.
Tiny networks often underfit rather than overfit, motivating methods that add training flexibility instead of further restricting the model.

50:00 Network Augmentation Temporarily Expands Training Capacity

Network augmentation temporarily adds channels or layers during training, then removes them so the deployed network retains its original compact size.
The augmented path shares the base network’s weights and supplies additional gradients, while the original task loss remains to ensure the base model works after the extra structure is removed.
Training can sample different augmented widths or depths while preserving all original model weights.
Experiments show gains for small models, including improved transfer to datasets such as Food-101, Flowers-102, and PASCAL VOC; on ResNet-50, added capacity can instead worsen validation accuracy through overfitting.

56:00 Combining Distillation and Augmentation for Efficient Models

Knowledge distillation transfers a teacher’s outputs or internal structure; network augmentation instead gives an underfitting student temporary training capacity.
The lecture reports that distillation and network augmentation can be combined to improve compact-model performance.
For a fixed compute budget, network augmentation improves object-detection accuracy; for a target accuracy, it can reduce required computation.
The lecture identifies applying network augmentation to compact LLMs and vision-language models as an open research direction.

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, MIT HAN Lab.