CMU AI Agents 2026: 3. Long Context Modeling for Agents
Overview
Graham Neubig explains why agent histories grow costly and unreliable, then surveys techniques for handling long contexts across model architecture, training, inference, caching, and compaction. He connects practical agent performance to concrete methods including linear attention and DeltaNet, sparse attention, RoPE extension, KV caching with vLLM and SGLang, and compaction policies that preserve important instructions.
Key takeaways
- Agent context cost grows rapidly because each call often reprocesses the accumulated history; OpenHands sessions averaged about 78,000 tokens, with tool results alone making up 37%.
- A million-token context window is a capacity claim, not evidence of reliable use: long-context retrieval benchmarks and agent tests must check whether the model retains and applies distant instructions.
- Hybrid architectures reduce long-sequence compute by combining local or recurrent-style methods such as Gated DeltaNet with sparse or dense global attention.
- KV caching can make repeated agent inference largely linear rather than quadratic, and cached input tokens may cost about one-tenth as much as uncached input tokens.
- Prompt-cache reuse depends on an unchanged prefix, so dynamic system prompts and frequent model switching can erase the cost benefits of caching.
- Compaction should protect initial instructions, preserve recent state, and retain access to full history; single-turn benchmarks can miss failures such as duplicate pull requests or forgotten mid-session constraints.
Chapters
- Each agent call adds messages, tool calls, and results to the history, so repeatedly processing the full history makes cumulative token work grow quadratically.
- An analysis of 1,500 OpenHands sessions found an average of about 78,000 tokens per session.
- In those sessions, tool results accounted for 37% of tokens, tool calls 20%, system prompts and tool descriptions 23%, and direct model replies only 2%.
- Prefill processes input tokens and builds their key-value states; decoding generates output tokens incrementally and extends the KV cache.
- Time to first token (TTFT) measures the wait before generation begins, while time per output token measures the pace of subsequent generation.
- Throughput and cost matter alongside latency, and interactive coding agents may prioritize TTFT and token speed more than overnight background agents.
- A million-token context window does not guarantee that a model can use evidence across the full million tokens.
- Needle-in-a-haystack tests place a fact inside longer documents, then ask the model to retrieve it; performance can degrade as context grows.
- Benchmarks such as RULER and HELMET test long-context capabilities, while forgotten user instructions can cause concrete agent failures such as deleting something the user said to preserve.
- Full attention makes pairwise comparisons across all tokens, leading to quadratic work; a sequence of one million tokens implies roughly a trillion token pairs.
- Many modern models combine local computation with periodic global attention rather than relying on full attention everywhere.
- Sliding-window attention limits each token to a fixed number of nearby chunks; information can still travel farther across layers as each chunk incorporates context from earlier chunks.
- Standard softmax attention compares queries against keys across the sequence, which makes its computation quadratic in sequence length.
- Linear attention removes softmax and uses associativity to accumulate key-value information into a state matrix that maps queries to values.
- Updating that matrix as new tokens arrive gives recurrent-style, linear-time processing, but loses some of softmax attention's ability to make selective comparisons among past tokens.
- DeltaNet compares the value predicted by its current state with the desired value, then updates the state to reduce that prediction error.
- A beta factor scales the update, while a decay factor discounts older state so distant context cannot keep influencing predictions indefinitely.
- Gated DeltaNet is used in Qwen; GLM and Kimi use more expressive variants with per-element decay rates, allowing parts of the state to persist for different durations.
- Fixed-pattern sparse attention attends only at selected intervals, reducing comparisons relative to full attention while retaining some global reach.
- Content-dependent sparse attention first scores compressed or projected representations, selects the top-k candidates, then performs full attention on those candidates.
- DeepSeek uses content-dependent sparse attention; the initial selection can be computed cheaply, though the full attention pass adds another step.
- Grouped-query attention shares keys and values across groups of query heads, reducing the number of stored KV heads and the memory required by the cache.
- Multi-query attention is the limiting case with one shared key-value set; grouping can reduce KV memory by roughly four or eight times.
- Multi-head latent attention reduces the dimensionality of keys and values, and can be combined with grouped-query attention for further KV-cache savings.
- Training at 128,000 tokens or beyond is difficult because relatively few internet documents contain that many coherent tokens.
- A typical approach starts with short-context pretraining, continues training on longer documents, then adapts to the target long context; packing unrelated documents does not teach cross-document coherence.
- Long-context data can include books, codebases, related document collections, patents, agent trajectories, and synthetic tasks.
- RoPE encodes relative token distance; some autoregressive models can also learn order from causal masking and local biases without explicit positional encodings.
- A model trained to length L has not learned how to handle token pairs separated by more than L positions, so extending its context window requires adaptation.
- Position interpolation stretches RoPE positions by the context-extension factor; YaRN varies the extension across frequencies rather than scaling every position uniformly.
- Open-model reports describe mixes of long documents, code repositories, issues, pull requests, scientific papers, technical reports, and synthetic long-context tasks.
- Training curricula may scale context lengths progressively, for example from 8K to 64K, 256K, and then one million tokens.
- Context parallelism distributes portions of a long sequence across devices, so each device computes key-value states for its assigned context segment.
- It becomes especially important for training contexts above roughly 32K tokens, alongside data parallelism and model-parallel methods such as tensor, pipeline, and expert parallelism.
- Specialized attention methods can create engineering bottlenecks: Graham Neubig found no readily available context-parallel kernel for Qwen's Gated DeltaNet in his attempt to train a smaller Qwen model.
- In an agent loop, prior messages and generated tokens can reuse their KV states; only new tool results and new output need fresh computation.
- Without caching, repeatedly processing an expanding history has quadratic cost; with caching, the repeated work is largely linear in sequence length.
- Cached input tokens can cost about one-tenth as much as uncached input tokens, so a 90% to 95% cache hit rate is a useful target for agent workloads.
- Output tokens can cost about five times as much as cached input tokens, making both cache hits and output volume important to total cost.
- vLLM's PagedAttention stores KV data in blocks that need not be contiguous in GPU memory, enabling flexible allocation, parallel sequence serving, and block-level eviction.
- SGLang's RadixAttention organizes cached prefixes in a radix tree so requests can reuse matching prompt history, a useful feature for long agent conversations.
- Multi-machine serving can route requests to the worker with the best matching KV cache; cache eviction time and routing quality both affect hit rates.
- GPU memory is precious, so serving systems may offload cached states to CPU memory, while providers can set different cache-retention windows.
- Prefix caching works only when the entire cached prefix is identical, so changing an early system message can invalidate reuse for the rest of the prompt.
- Do not update the system prompt with the current date or time on every agent step; place changing information in the latest user message or tool result instead.
- Switching models mid-conversation can also reduce cache reuse, so model routing decisions should account for the cost of rebuilding the new model's prompt state.
- Compaction replaces a long interaction history with a bounded working state intended to preserve future behavior; common policies protect initial instructions, summarize older events, and retain the recent tail.
- Compaction can be triggered by a token threshold, a provider's context limit, or a manual command such as slashcompact; summaries may be free-form or structured.
- A benchmark can miss real failures: OpenHands testing found that an agent could forget an already-created pull request and submit duplicates, or lose user instructions given mid-conversation.
- Repeated compaction can compound information loss, so preserve access to the full history when possible and test compaction in realistic multi-turn workflows, not only on single-turn benchmarks.
Transcript excerpt
0:07 Okay. Uh hello everyone. Can you hear me in the back? All good. Okay. So, uh well, welcome back everyone. Am I uh echoing? All good. Okay. So, um, today I'd like to talk about context management for long context agents. And I at first thought about titling this as like context management for long context LLMs because most of the stuff I'm going to be talking about today is kind of more general LLM stuff, but long context is super super important for when we're working on agents and it's not necessarily covered uh, very well in all of the LLM prerequisite classes. So, I'm going to cover it in a fair amount of detail today. I'm also going to try to go through kind of the long context
0:55 architectures that are used in many of the kind of state-of-the-art open uh models so you can kind of see the best practices that everybody is using in order to handle a very long context. So first I'd like to talk a little bit about um how long an agent context can grow. So basically the input is quadratic uh from the uh or the number of token accesses that you do is quadratic uh based on the linear history growth. So um if each call uh you make for an agent takes about you know five tokens and let's say you start out with a an initial system prompt of one token uh you'll get six tokens in the first call
1:42 11 16 uh 21 and 26. you know, this will increase and increase and increase. Um, and so by the time you have gotten to the end, you will, if you do things completely naively, you will have had to process ADK tokens uh for these five calls. And you know, this can continue
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Graham Neubig.