EfficientML.ai Lecture 15 - Long-Context LLM (MIT 6.5940 Fall 2026)
Watch on YouTube →
Overview
Long-context LLMs face two linked challenges: extending a model's usable context and controlling the memory and latency costs of processing it. The lecture explains RoPE scaling and LongLoRA for context extension, Needle-in-a-Haystack and LongBench for evaluation, and efficient inference methods including StreamingLLM, DuoAttention, Quest, and LServe, then introduces Mamba and hybrid Transformer architectures.
Key takeaways
- LongLoRA makes long-context fine-tuning more accessible by combining shifted local sparse attention during training with LoRA adapters, then restoring full attention at inference.
- A model's fluent output does not prove it used a long prompt: the lost-in-the-middle effect makes position-sensitive retrieval tests such as Needle-in-a-Haystack important.
- StreamingLLM bounds KV-cache memory by retaining initial attention-sink tokens and a recent window, but it cannot retrieve information after that information has been evicted.
- DuoAttention divides heads between full-cache retrieval and bounded-cache streaming, reducing memory and decoding cost while preserving dedicated heads for long-range retrieval.
- Quest makes token importance query-dependent: it retains the full KV cache but fetches only pages estimated to matter for the current query, achieving near-perfect needle retrieval at roughly 1% of the full token budget.
- Mamba uses selective state-space updates and parallel scan for linear-time sequence processing; hybrid designs such as Jamba combine that efficiency with Transformer retrieval.
Chapters
- Rotary Position Embedding (RoPE) treats pairs of hidden dimensions as coordinates in a complex plane and rotates them according to token position and dimension.
- Because RoPE encodes relative position through rotation, scaling its position frequencies can fit longer sequences into a familiar angular range.
- Position interpolation can, for example, compress positions to support a context window about 2× the original length.
- LongLoRA targets efficient fine-tuning for long contexts, avoiding the quadratic attention cost of full long-sequence training.
- It splits attention heads between local group attention and a cyclically shifted local pattern, allowing information to move between neighboring groups across layers.
- LoRA adapters on attention projections reduce training cost, while inference returns to full attention to preserve accuracy.
- The lecture describes extending a 7B-parameter model to a 32,000-token context on a single 8-GPU node, rather than relying on a large training cluster.
- LongLoRA reduces training time and memory compared with full-attention LoRA while more closely matching full fine-tuning perplexity at longer sequence lengths.
- Fine-tuning normalization and embedding layers helps close the performance gap that remains with LoRA alone, even at higher adapter ranks.
- Passkey retrieval tests show that LongLoRA maintains accuracy near its target context length; positional interpolation can extend the useful range further.
- Long-context models often show a U-shaped retrieval pattern: they find information near the start or end of a document more reliably than information in the middle.
- A fluent answer can still omit important evidence, so language quality alone does not demonstrate that a model used the full context.
- Needle-in-a-Haystack and LongBench test different aspects of long-context ability, from controlled retrieval to broader real-world tasks.
- Needle-in-a-Haystack varies context length and the depth of a hidden fact, revealing where retrieval succeeds or fails.
- LongBench evaluates 21 datasets across question answering, summarization, few-shot learning, coding, and multilingual tasks, including English and Chinese.
- LongBench uses contexts above 13,000 tokens and assesses broader comprehension than synthetic passkey retrieval alone.
- For a LLaMA 2 70B model using multi-head attention, batch size 16 and a 4K sequence can require roughly 60 GB of KV-cache memory.
- KV-cache use grows with both batch size and sequence length, and can exceed the storage required for model weights at long contexts.
- Streaming generation and extended conversations can exhaust device memory or exceed the context lengths seen during pretraining, degrading output quality.
- A sliding window limits KV-cache growth by evicting old tokens, but perplexity can rise sharply when the first tokens are discarded.
- Repeatedly rebuilding a sliding-window cache preserves more quality but recomputes substantial work for each new token.
- StreamingLLM aims to retain the quality of recomputation with bounded memory and the low per-token cost of a rolling cache.
- Attention sinks are initial tokens that receive substantial attention across heads and layers, even when their semantic content is unimportant.
- Because softmax attention weights must sum to one, initial positions provide a consistent place for attention that has no stronger target.
- StreamingLLM keeps a small set of sink tokens plus a recent-token window, allowing fixed-size KV-cache memory while supporting continuous generation.
- Position IDs must be updated so retained sink and window tokens form a continuous position sequence; this method does not preserve access to evicted information.
- DuoAttention divides attention heads into dense retrieval heads, which retain all KV entries, and streaming heads, which retain sinks and a recent window.
- The split targets long-context inference bottlenecks: KV storage for every token and decoding latency that grows with context length.
- A lecture demo reports processing 1.84 million tokens with 49.3 GB of memory, and up to 3.3 million tokens on a single A100.
- During training, a gate mixes full-attention and streaming-attention outputs; larger gate values indicate heads that need full retrieval.
- A distillation objective penalizes reliance on dense attention, then classifies heads by their learned gate values.
- Only about 1,000 scalar gates need tuning for a model with 32 layers and 32 heads, making the process relatively lightweight.
- The lecture reports over 2× memory reduction and decoding speedup for MHA models, with over 60% memory savings and 1.5× decoding speedup for GQA models.
- Quest keeps the full KV cache but reduces data movement by selecting a small set of relevant KV pages for each new query.
- It estimates page importance using compact key statistics, including element-wise minimum and maximum values, then computes exact attention only for selected pages.
- On Needle-in-a-Haystack, Quest achieves nearly perfect retrieval with a token budget around 1% of the full context.
- Query-aware selection avoids permanently discarding a token that seems irrelevant now but may become essential to a later question.
- LServe represents streaming and retrieval attention as block-sparse patterns across prefilling and decoding, with separate KV-page layouts for streaming and dense heads.
- Large physical pages improve memory efficiency, but can hurt selection accuracy; LServe uses smaller logical pages inside larger physical pages to balance the two.
- The selector scores logical pages and assigns each physical page the maximum score among its logical pages, preserving important regions while fetching hardware-friendly blocks.
- A reusable page selector shares selection results across consecutive queries that attend to the same KV pages, reducing selection overhead.
- Mamba replaces self-attention's token-to-token communication with a state-space model that updates a fixed-size recurrent state in linear time.
- Input-dependent state-space parameters let Mamba selectively retain, forget, or expose information; parallel scan supports efficient training despite recurrent inference.
- Hybrid models such as Jamba interleave Transformer attention and Mamba layers, combining precise retrieval with improved memory and throughput.
- The described hybrid configuration supports a 256K-token context on a single 80 GB GPU.
Transcript excerpt
0:01 Uh okay, good afternoon everyone. Uh let's get started for today's lecture. And today we are going to talk about uh uh long context language models. Um which really matters when we want to use a model to understand very uh long documents uh understand for example a book a loan technical document uh codebase etc. So the challenge here is that attention the KB cache can be uh very expensive as a sequence grows and we will study techniques that can extend the context lens uh evaluate whether models truly use a long context and reduce the memory and latency of the long context inference. So here is a overview of today's uh lectures contents. So we will u revisit
0:52 what is u uh rotary position embedding. uh we have discussed it uh several lectures uh earlier and we will also talk about uh uh some work on uh extends the language model to longer context with some finetuning techniques for examples Laura and we will also talk about some benchmarks in evaluating long context performance for large function models uh such as uh needle in a haststack benchmark and the lone bench after that we will talk about some efficient attention mechanisms to reduce uh overall inference time for large language models uh particularly some sparse attention uh techniques. And finally we will briefly introduce some uh architectures that is
1:41 beyond traditional transformers uh especially uh uh linear models uh for examples state space models u mamba and some other variants. Uh so first of all let's uh have a very quick uh recap of the uh rotary position embeddings that we talked uh previously. So if you still remember that in uh in the rope we h we will divide the uh hidden state of the lang language model
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, MIT HAN Lab.