Save this video free

CMU AI Agents 2026: 2. Tool Use for Language Model Agents

Graham Neubig · 57:22 · Watch on YouTube

Part of Carnegie Mellon University 11-768. See all 14 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

Graham Neubig explains how language model agents use tools to access fresh information, perform reliable computation, and act on external systems, making tool use a core distinction between an agent and a text-only model. He traces the full tool-calling stack, from model-specific formats and schema validation through grammar-constrained decoding, REST and MCP integrations, parallel execution, and evaluation, highlighting CodeAct, XGrammar, and the Berkeley Function Calling Leaderboard.

Key takeaways

Chapters

0:00 Why Language Model Agents Need External Tools
6:07 Common Agent Tools: Search, Code, Images, and APIs
10:20 CodeAct Makes Programs a Meta-Tool
16:09 Tool-Call Tokens, Chat Templates, and Model-Specific Formats
23:29 Dispatching Tool Calls and Matching Results by ID
27:07 XGrammar and Constrained Decoding for Valid Tool Arguments
35:40 REST APIs and FastAPI as Agent Tool Interfaces
39:11 MCP Separates Agent Credentials from Upstream Secrets
45:16 Parallel Tool Calling Reduces Agent Latency
48:53 Evaluating Tool Choice, Arguments, and Reliability
51:32 Why Tool-Call Errors Differ Across Model Providers

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:00 Okay. So, hi everyone. Uh, welcome to class number two. Today, uh, today I'll be talking about tool use for language model agents. And, uh, this is, you know, the the most fundamental thing about agents that, uh, you know, makes them different from a language model. Um so the basic idea here is uh that we're going to be um a tool is basically an interface through which a language model can invoke an external computer program. And uh this is a language model cannot do this on its own because it's fundamentally a token predicting machine. But this enables you to do you know many many different things. So you

0:48 know you can search, you can call a calculator, you can call a browser, um you can call Python, you can call arbitrary APIs and so uh the language model kind of as you'll see the method for tool calling is standardized but what you do downstream with the tool call can be very different. So there's basically two uh benefits of tools and all of this is from one of the readings that I I posted on the website if you want to go back and read more about it. Uh this is a survey that uh I we wrote uh together Daniel and I and Zoro Wong, one of our our students. Um and the first thing is extending the ability of language models. So this is giving language models the ability to do

1:35 something that they fundamentally cannot do. And so this could be like accessing information or actions that are outside the language model's um you know action space. The second thing is uh facilitating the language model. And so what I mean by this is this is something that fundamentally the language model

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Graham Neubig.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.