CMU AI Agents 2026: 12. RL Systems
Overview
Reinforcement learning systems for language-model agents combine specialized inference and training engines, because autoregressive decoding, KV-cache use, activation storage, and backpropagation need different optimizations. The lecture explains how frameworks such as vLLM, SGLang, Megatron, FSDP2, SkyRL, and Miles coordinate rollouts, training, and weight updates, then examines agent-specific choices including prefix-preserving trajectories, token-level data transfer, asynchronous execution, and service-based architecture. Practical techniques include prefix caching, continuous batching, activation checkpointing, recorded MoE routing, delta weight synchronization, and multi-tenant LoRA training.
Key takeaways
- Separate inference and training engines because autoregressive decoding needs KV-cache capacity and token-by-token scheduling, while backpropagation needs activation storage and parallel sequence computation.
- Agent rollouts contain extensive shared context, so prefix caching in SGLang or vLLM and tree-based training methods can reduce repeated computation across prompts and trajectory branches.
- Preserve the exact token IDs and routing decisions produced during inference; text re-tokenization or different MoE expert choices can make training operate on a different policy than the one that generated the data.
- Asynchronous RL helps with uneven agent task durations by training on completed rollouts while other tasks continue, but weight updates require pausing generation and refreshing stale KV caches.
- Service-based architectures with standard APIs decouple training, inference, tools, and environments, enabling independent scaling, hardware mixing, online updates, and concurrent multi-LoRA workloads.
Chapters
- Core training operations include model forward passes, loss and gradient computation, and optimizer steps.
- Agent RL adds environment interaction, action execution, and reward evaluation to the standard training loop.
- A framework must connect rollout generation with training rather than simply call PyTorch or JAX APIs.
- Inference engines such as vLLM and SGLang optimize sequential token decoding, while training engines such as Megatron and FSDP2 optimize parallel forward and backward passes.
- Inference can devote memory to KV caches, while training must retain activations and computation graphs for backpropagation.
- Separating the engines enables workload-specific techniques such as continuous batching for inference and activation checkpointing for training.
- A Llama 3 8B example estimates that 64 sequences of 32,000 tokens can require about 256 GB of BF16 KV-cache memory.
- Agent tasks often reuse system prompts, task prompts, and earlier trajectory context, making prefix caching valuable across requests.
- SGLang RadixAttention and vLLM automatic prefix caching reuse shared prefix computations.
- Continuous batching schedules new requests as decoding slots free up, while segmented prefill prevents long prompts from blocking decode work.
- A smaller draft model proposes several tokens for a larger model to verify using rejection sampling, preserving the target policy's output distribution.
- The draft model's acceptance rate can fall as RL updates move the main policy away from its initial behavior.
- SFT or distillation can keep the draft model aligned with the evolving policy and sustain speculative decoding gains.
- Separating prefill and decode onto different devices can match each workload to hardware suited to its compute pattern.
- Activation checkpointing stores selected layer inputs and recomputes discarded activations during backpropagation, trading about 33% more computation for lower memory use.
- Training batches with shared prefixes can avoid repeated forward computation, but must also manage activation memory and backward passes.
- Snowflake's Arctic RL work and the Zorro system use dynamic tree attention to process shared prefixes across branches.
- Depth-first traversal can release activations after each branch's backward pass, keeping memory near that needed for one root-to-leaf path.
- Data parallelism replicates a model across GPUs and splits the global batch into smaller batches that fit memory limits.
- Pipeline parallelism assigns different model layers to stages, while microbatches keep stages working during forward and backward passes.
- Tensor parallelism splits layer computations but requires frequent communication, so it is usually favored within a fast-connected GPU node.
- Expert parallelism distributes mixture-of-experts components, and context parallelism splits long sequences while transferring attention keys and values between ranks.
- Synchronizing a trillion-parameter BF16 model as full weights can mean transferring about 2 TB per update; sending changed parameters may reduce transfers to roughly 20 to 30 GB.
- Small reward-driven updates may leave many BF16 parameters unchanged between updates, making delta synchronization practical.
- For mixture-of-experts models, inference and training can choose different experts because of small implementation differences.
- Recording rollout routing decisions and replaying them during training keeps the training forward pass aligned with the sampled policy.
- Megatron and FSDP2 are common training backends, while vLLM and SGLang are widely used for inference.
- Frameworks including SkyRL, Miles, PrimeRL, and Slime connect these lower-level engines to RL algorithms and rollout workflows.
- Hosted services let users specify models, data, and reward functions without managing GPU infrastructure directly.
- Model-specific recipes, such as those offered by Miles, provide a tested starting configuration to adapt.
- Appending actions and observations into one continuous trajectory can avoid repeatedly prefilling the same context; periodic context summaries split the trajectory and add computation.
- When trajectories are split into separate training examples, Dynamic Tree Attention can reuse shared prefix computations.
- Chat templates that remove prior reasoning or otherwise rewrite context break prefix continuity; RL frameworks provide custom renderers to preserve it.
- Passing token IDs from inference to training avoids text re-encoding mismatches, which can change tokens and cause policy drift or unstable loss.
- Agent tasks have highly variable durations, so asynchronous execution can train on completed short rollouts instead of waiting for the slowest trajectory.
- Training and inference can run concurrently on separate GPU allocations, with rollouts often receiving more resources when long decoding chains are the bottleneck.
- An example allocation is a 3:1 inference-to-training ratio, such as six inference nodes and two training nodes.
- SGLang and vLLM support pausing or retracting generation; retracting clears stale KV cache so it can be recomputed under updated weights.
- A master agent can delegate work to subagents that may delegate further, creating multi-level execution trees.
- Agent harnesses can use context compression, changing how much history is available to later actions.
- RL data collection must decide whether to treat an entire delegation tree as one rollout or collect subagent paths separately.
- A tool-facing proxy can mimic chat-completion or response APIs, route calls to an inference engine, and record probabilities and rollout data without modifying each agent tool.
- SPMD frameworks run the same program on every GPU and use rank-based branches to assign roles, a simple pattern that can become inflexible as systems grow.
- A controller-based design connects training, inference, weight synchronization, and environment services through standard protocols such as MCP.
- Separating components lets teams maintain specialized services independently while keeping orchestration lightweight.
- Tinker, SkyRL, and Miles support a common training interface that can make it easier to switch between hosted and self-managed systems.
- A service architecture can scale to multiple inference servers and heterogeneous hardware, and can assign different trainers or models to separate agents.
- Online training can use data collected from deployed tools; Cursor has described updating Composer checkpoints on a roughly five-hour cadence.
- Multi-tenant systems such as Tinker run multiple LoRA adapters concurrently, improving overall throughput compared with processing adapter jobs sequentially.
Transcript excerpt
0:00 Today, we will talk about reinforcement learning systems. This is a very interesting but broad topic. My goal today is to provide an overview of the things you will encounter if you use reinforcement learning frameworks or systems to train agents, or if you want to modify some of these systems . So, we will start with an overview, and then we will talk about some things specific to agent-oriented reinforcement learning. We will then also discuss some of the system design options that these frameworks might take . In the previous two lectures , we talked about updating the policy gradient where
0:47 we have a weighted target that is very similar to Supervised Fine-Tune (SFT) but with a feature-based weight , right? So, one question is: what processes should be available within a reinforcement learning framework if we want to train such models? What are the basic priorities or processes that we might need in such a framework? Does anyone want to mention a few of them or raise their hand? The conclusion is yes. So, a type of forward pass. Are there others? There's a hint here. We have the
1:35 gradation. Calculating the loss. Back passes, yes. And there's a third one, we have these forward passes, don't we? We need it not only to account for the loss, but also because in reinforcement learning we are sampling our own paths or results, right? Therefore, we need to make forward passes for that. We want to calculate the gradient. Then we have a third type of basic operation, which is the optimizer step, where after calculating the update for the models , we want to update the actual model weights. But for agent- directed reinforcement learning , there are
2:22 other things we need, such as the environment, right? In
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Graham Neubig.