EfficientML.ai Lecture 12 - Transformer and LLM (MIT 6.5940 Fall 2026)

Overview

The lecture explains how Transformer components, including self-attention, feed-forward networks, positional encodings, and KV caching, underpin modern large language models. It compares encoder and decoder architectures, details techniques such as RoPE, GQA, and SwiGLU, and connects scaling and in-context learning to model families including BERT, GPT, T5, and LLaMA, then surveys multimodal models and mixture-of-experts routing.

Key takeaways

Causal self-attention lets GPT-style models train on next-token prediction and generate autoregressively, while BERT's bidirectional attention supports masked-token reconstruction from both sides.
The KV cache trades memory for decoding speed: it avoids recomputing past keys and values, but its storage grows with batch size and context length and can reach roughly 160 GB in the lecture's 70B-model example.
Grouped-query attention reduces KV-cache size by sharing keys and values across query-head groups; an eight-query group can deliver about an 8× reduction while preserving more quality than single-key-head MQA.
RoPE encodes relative token distance through rotations of query and key dimensions, and position interpolation can extend a pretrained context window, with the lecture citing a LLaMA example expanded from 2K to about 32K tokens.
In-context learning adapts a sufficiently capable model through instructions or demonstrations in its prompt, without gradient updates or a separate fine-tuned model for every task.
Multimodal architectures connect perception and action to language models in different ways: Flamingo uses cross-attention for visual input, while RT-2 represents robot commands as generated tokens.

Chapters

0:00 LLM Applications: ChatGPT, Translation, Coding, and Multimodal Models

ChatGPT's public debut in late 2022 helped make chatbot-style large language models widely accessible.
LLMs support machine translation and coding assistants such as GitHub Copilot, Cursor, and Codex.
Multimodal models extend language-model capabilities to image, video, and audio understanding and generation.

4:00 Why Transformers Replaced RNNs and Convolutional Language Models

RNNs compress earlier inputs into a fixed-size hidden state, but long-range information must pass through many sequential steps.
RNN dependencies limit parallel computation during training and inference on GPUs.
Convolutional models parallelize across positions, but long-range context requires larger receptive fields or additional layers.

9:40 Transformer Inputs: Tokenization, Embeddings, and Layer Components

Tokenizers split text into model tokens, so token count, not word count, determines sequence length.
A learned embedding lookup maps each token to a dense vector instead of a vocabulary-sized sparse one-hot representation.
Transformer blocks combine attention, feed-forward networks, layer normalization, residual connections, and positional information.

15:20 Scaled Dot-Product Attention and Multi-Head Attention

Attention projects hidden states into queries, keys, and values; query-key dot products estimate which values are relevant.
Scaling scores by the square root of the head dimension helps keep softmax inputs well behaved.
Multi-head attention learns several query-key-value projections in parallel, allowing heads to capture different relationships.
Full attention has quadratic sequence-length computation, commonly expressed as O(n²).

20:00 Causal Masks Separate GPT Generation from Bidirectional Attention

Causal attention masks future positions, allowing each GPT token to use only itself and earlier tokens.
Unmasked global attention lets each token inspect the full input, a pattern suited to encoder tasks such as BERT.
Different attention heads can learn distinct patterns, including local relationships and longer-distance dependencies.

23:00 Transformer FFNs, LayerNorm, and Residual Connections

A basic feed-forward network expands a hidden dimension D to 4D, applies a nonlinear activation, and projects back to D.
Attention mixes information across tokens, while the FFN adds nonlinear transformations within each token representation.
LayerNorm normalizes each token's features and uses learned scale and shift parameters, gamma and beta.
Pre-LayerNorm places normalization before attention and FFN sublayers and is favored in many modern LLMs for training stability.

27:45 Position Information and the Original Transformer’s Translation Results

Positional information lets the model distinguish identical tokens at different locations and reason about token distances.
Absolute positional embeddings add a position-specific vector to each token embedding; relative methods encode distances between tokens.
The original Transformer in “Attention Is All You Need” achieved higher BLEU scores than earlier translation systems while using less training computation.

31:10 Encoder-Decoder, BERT, and GPT Architectures

Encoder-decoder models such as T5 encode a source sequence and condition target-token generation on its representation.
BERT uses bidirectional encoder attention and masked-language modeling, reconstructing masked tokens from left and right context.
GPT uses decoder-only causal attention and next-token prediction; large models can respond to new tasks from instructions or examples in a prompt.
Translation and summarization fit naturally into text-to-text encoder-decoder tasks, while GPT-style models generate autoregressively.

36:00 Relative Position Encoding with ALiBi

Absolute position embeddings are simple to use but can struggle when inference sequences exceed the positions seen during training.
ALiBi, or Attention with Linear Biases, adds a distance-dependent bias directly to attention scores.
Different attention heads use different bias slopes, allowing some heads to favor nearby tokens while others retain more distant context.

40:00 RoPE: Rotary Position Embeddings and Context Extension

RoPE rotates pairs of query and key dimensions by position-dependent angles, making their attention interaction depend on relative distance.
Implementations can apply the rotations efficiently with elementwise operations rather than explicit large matrix multiplications.
Position interpolation rescales rotary positions to extend context; the lecture cites extending a LLaMA model from 2K to about 32K tokens without additional pretraining.
NoPE is another design that can let causal attention structure provide positional cues without explicit position embeddings.

48:00 KV Cache: Faster Autoregressive Decoding at a Memory Cost

A KV cache stores previously computed keys and values so each new token does not recompute them for the full prefix.
Caching reduces total generation work from roughly O(n³) without reuse to O(n²) across an n-token output sequence.
Cache memory grows with layers, sequence length, batch size, and key-value dimensions.
For a 70B-class example, the lecture estimates about 2.5 MB of KV cache per token; batch size 16 and 4K tokens can require roughly 160 GB.

54:30 GQA, MQA, and SwiGLU Reduce LLM Inference Costs

Multi-query attention shares one key-value head across query heads, greatly reducing cache size but potentially harming model quality.
Grouped-query attention shares each key-value head across a group of query heads; a group size of eight can reduce KV-cache storage by about 8×.
The lecture describes GQA as a quality-memory compromise and notes that LLaMA 2 70B uses it.
SwiGLU uses separate up and gate projections followed by elementwise multiplication and a down projection, with dimensions adjusted to keep parameter count comparable to a standard FFN.

1:01:00 Emergent Capabilities, In-Context Learning, and LLaMA Scaling

Some benchmark capabilities, including arithmetic and word unscrambling examples, rise sharply after models reach certain scales, motivating the term emergent abilities.
Zero-shot prompts provide task instructions; few-shot prompts add input-output demonstrations without updating model weights.
The LLaMA family combines decoder-only pre-normalization, SwiGLU, and RoPE; LLaMA 2 trained on roughly two trillion tokens and its 70B model uses GQA.
LLaMA 3 scaled training to more than 15 trillion tokens, and its flagship model has over 400 billion parameters; post-training included supervised fine-tuning, rejection sampling, and DPO.

1:08:20 Multimodal Models: Flamingo, PaLM-E, and RT-2

Flamingo connects visual features to a language model with cross-attention and compresses image features into a fixed set of tokens using a Perceiver Resampler.
A learned gate initialized at zero gradually controls visual input to a frozen language model during training.
Flamingo supports multimodal in-context learning and follow-up dialogue that refers back to images in the conversation.
PaLM-E represents images, robot states, and sensor data as tokens, while RT-2 generates robot control actions as output tokens and transfers knowledge from web data to robotics.

1:14:25 Mixture-of-Experts Routing and Capacity Trade-Offs

A router selects only some expert FFNs for each token instead of activating every parameter in the model.
A model with 30B total parameters and 3B active parameters per token can increase capacity without paying the full dense-model inference cost per token.
Expert capacity factors limit tokens per expert; higher capacity reduces dropped tokens but may leave more unused capacity.

Transcript excerpt

0:02 Okay. Uh let's get started for today's lecture. So we will talk about uh transformers and large language models. Uh we will have um just introduction of the transformer architecture and uh some application of large language models and we will talk more uh more details like uh how do you do the deployment, how do you do the post training, how do you uh apply large language models to different kind of applications in uh future lectures. So today uh we will just dive into the overall architecture and have a very high level overview of the large language model architecture and its applications. So to begin with um we all know that uh CH GBT come out in late 2022 and Sam's uh chatbots large language models

0:54 becomes very prevalent and uh accessible to almost everyone and at the very beginning it's just a a chatbot and you can type in something like some just some easy questions and get some response from the uh model at that time. uh later um we can see we see that it is applied to more and more application domains. So machine translation is very familiar uh another very familiar application to everyone and a language model can take the taxing uh from one lang language and uh generate the corresponding tax in another language uh while preserving the original meaning and the context. So with more and more advanced language language models you may feel that nowadays machine

1:41 translators are performing much better than let's say five years ago or 10 years ago right and another very important application is uh uh use large language model to assist people do coding tasks and this is a very uh early uh example for uh AI assistant coding uh said it uh which is

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, MIT HAN Lab.