Save this video free

CMU AI Agents 2026: 3. Long Context Modeling for Agents

Graham Neubig · 1:15:39 · Watch on YouTube

Part of Carnegie Mellon University 11-768. See all 14 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

Graham Neubig explains why agent histories grow costly and unreliable, then surveys techniques for handling long contexts across model architecture, training, inference, caching, and compaction. He connects practical agent performance to concrete methods including linear attention and DeltaNet, sparse attention, RoPE extension, KV caching with vLLM and SGLang, and compaction policies that preserve important instructions.

Key takeaways

Chapters

0:00 Why Agent Contexts Grow So Quickly
6:00 Inference Phases and Serving Metrics
12:00 Long Context Capacity Is Not Long Context Competence
16:00 Hybrid Attention Architectures and Sliding Windows
20:00 Linear Attention Replaces the Growing KV History with State
25:00 DeltaNet Adds Error Correction and Forgetting
31:00 Sparse Attention Selects Which Past Tokens to Inspect
34:00 Grouped Query Attention and Latent KV Compression
36:00 Long-Context Training, Positional Encodings, and Data
43:00 Extending RoPE and Building Long-Context Training Sets
49:00 Context Parallelism for Long-Sequence Training
53:00 KV Caching Reduces Repeated Agent Inference
57:00 vLLM, SGLang, and Cache-Aware Serving
1:04:00 Preserve Prefixes to Avoid Breaking Prompt Caches
1:07:00 Context Compaction Must Preserve Instructions and State

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:07 Okay. Uh hello everyone. Can you hear me in the back? All good. Okay. So, uh well, welcome back everyone. Am I uh echoing? All good. Okay. So, um, today I'd like to talk about context management for long context agents. And I at first thought about titling this as like context management for long context LLMs because most of the stuff I'm going to be talking about today is kind of more general LLM stuff, but long context is super super important for when we're working on agents and it's not necessarily covered uh, very well in all of the LLM prerequisite classes. So, I'm going to cover it in a fair amount of detail today. I'm also going to try to go through kind of the long context

0:55 architectures that are used in many of the kind of state-of-the-art open uh models so you can kind of see the best practices that everybody is using in order to handle a very long context. So first I'd like to talk a little bit about um how long an agent context can grow. So basically the input is quadratic uh from the uh or the number of token accesses that you do is quadratic uh based on the linear history growth. So um if each call uh you make for an agent takes about you know five tokens and let's say you start out with a an initial system prompt of one token uh you'll get six tokens in the first call

1:42 11 16 uh 21 and 26. you know, this will increase and increase and increase. Um, and so by the time you have gotten to the end, you will, if you do things completely naively, you will have had to process ADK tokens uh for these five calls. And you know, this can continue

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Graham Neubig.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.