Save this video free

CMU AI Agents 2026: 12. RL Systems

Graham Neubig · 1:06:27 · Watch on YouTube

Part of Carnegie Mellon University 11-768. See all 14 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

Reinforcement learning systems for language-model agents combine specialized inference and training engines, because autoregressive decoding, KV-cache use, activation storage, and backpropagation need different optimizations. The lecture explains how frameworks such as vLLM, SGLang, Megatron, FSDP2, SkyRL, and Miles coordinate rollouts, training, and weight updates, then examines agent-specific choices including prefix-preserving trajectories, token-level data transfer, asynchronous execution, and service-based architecture. Practical techniques include prefix caching, continuous batching, activation checkpointing, recorded MoE routing, delta weight synchronization, and multi-tenant LoRA training.

Key takeaways

Chapters

0:00 RL Frameworks Coordinate Sampling, Training, and Environment Feedback
5:00 Why vLLM Inference and Megatron Training Use Separate Engines
10:00 Prefix Caching and Continuous Batching Improve Agent Rollouts
17:00 Speculative Decoding Must Adapt as the RL Policy Changes
21:00 Activation Checkpointing and Dynamic Tree Attention Save Training Memory
25:00 Data, Pipeline, Tensor, and Expert Parallelism Fit Large Models on GPUs
29:00 Delta Weight Sync and Recorded Routing Reduce RL Engine Mismatch
35:00 Choosing RL Frameworks and Their Inference and Training Backends
37:00 Preserve Agent Trajectory Prefixes and Exact Token IDs During Training
47:00 Asynchronous RL Overlaps Long Agent Rollouts with Training
51:00 Agent Harnesses Add Delegation and Trajectory-Collection Choices
53:00 Tool Proxies and Service Architecture Decouple RL Components
59:00 Service-Based RL Enables Scaling, Online Updates, and Multi-LoRA Training

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:00 Today, we will talk about reinforcement learning systems. This is a very interesting but broad topic. My goal today is to provide an overview of the things you will encounter if you use reinforcement learning frameworks or systems to train agents, or if you want to modify some of these systems . So, we will start with an overview, and then we will talk about some things specific to agent-oriented reinforcement learning. We will then also discuss some of the system design options that these frameworks might take . In the previous two lectures , we talked about updating the policy gradient where

0:47 we have a weighted target that is very similar to Supervised Fine-Tune (SFT) but with a feature-based weight , right? So, one question is: what processes should be available within a reinforcement learning framework if we want to train such models? What are the basic priorities or processes that we might need in such a framework? Does anyone want to mention a few of them or raise their hand? The conclusion is yes. So, a type of forward pass. Are there others? There's a hint here. We have the

1:35 gradation. Calculating the loss. Back passes, yes. And there's a third one, we have these forward passes, don't we? We need it not only to account for the loss, but also because in reinforcement learning we are sampling our own paths or results, right? Therefore, we need to make forward passes for that. We want to calculate the gradient. Then we have a third type of basic operation, which is the optimizer step, where after calculating the update for the models , we want to update the actual model weights. But for agent- directed reinforcement learning , there are

2:22 other things we need, such as the environment, right? In

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Graham Neubig.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.