CMU AI Agents 2026: 2. Tool Use for Language Model Agents
Overview
Graham Neubig explains how language model agents use tools to access fresh information, perform reliable computation, and act on external systems, making tool use a core distinction between an agent and a text-only model. He traces the full tool-calling stack, from model-specific formats and schema validation through grammar-constrained decoding, REST and MCP integrations, parallel execution, and evaluation, highlighting CodeAct, XGrammar, and the Berkeley Function Calling Leaderboard.
Key takeaways
- Tool use both extends language models beyond their frozen knowledge and makes achievable tasks more reliable or efficient, as when web search supplies current evidence and a calculator replaces token-by-token arithmetic.
- CodeAct treats code as a meta-tool for coordinating APIs, improving success and reducing interaction turns, but its expressive power requires sandboxing against infinite loops, compromised dependencies, and excessive resource use.
- Tool-call formats differ across Qwen, Mistral, and DeepSeek, so applications must use model-compatible chat templates and parsers rather than assuming one serialization works for every model.
- XGrammar-style constrained decoding can enforce schema-valid tool-call syntax and arguments during generation, but it cannot guarantee semantic correctness or completion when the token budget runs out.
- MCP can reduce credential exposure by keeping sensitive upstream keys, such as GitHub tokens, on the MCP server and providing the agent with a separate credential.
- Independent tool calls can run concurrently, and reinforcement learning incentives for shorter task completion have helped newer models use parallel tool calling more effectively.
Chapters
- A language model predicts tokens, so it cannot independently access information newer than its training or change external systems.
- Tools extend model capabilities, such as web search for current facts, and facilitate tasks the model can do but a tool handles faster, such as seven-digit multiplication with a calculator.
- Tools can retrieve fresh evidence, execute exact computations, expose private user state, and trigger actions through APIs.
- Modern ChatGPT uses iterative tool calls, including web search, code execution, image generation, PDF creation, and integrations such as Slack.
- Web retrieval is now a routine form of retrieval-augmented generation, grounding answers in current sources.
- Image generation requires translating a user's request into a more detailed prompt for the image-generation API.
- Custom APIs can perform actions such as creating an Instacart grocery cart.
- A set of individual APIs handles one call or a parallel group of calls at a time, while code can express sequences, loops, conditionals, and library operations.
- CodeAct uses a program to coordinate repeated operations, such as comparing smartphone prices, exchange rates, and taxes across the US, Japan, Germany, and India.
- The CodeAct results Neubig describes show higher task success and fewer interaction turns, including on tasks not traditionally classified as coding.
- Code execution brings risks such as infinite loops, compromised libraries, broad permissions, and higher CPU, memory, and disk use, so it requires sandboxing and constraints.
- A model can produce user-facing text and a tool call in the same completion; reasoning models may also separate hidden thinking from the visible response and call.
- Tool-call tokens are commonly introduced during mid-training rather than necessarily being present in initial internet-scale pretraining.
- Tool definitions provide a function name and parameter schema, such as a get_weather function requiring a city string.
- Qwen, Mistral, and DeepSeek use different call formats, so inference and fine-tuning pipelines must apply the correct chat template and parser.
- An agent registers tools by name, parses a model call, validates its arguments, dispatches it to a Python or TypeScript function, and returns an observation.
- Failures can happen during argument validation or execution, such as syntactically accepted Python that raises an error when run.
- Each tool call needs a matching result ID, especially when several calls run in parallel.
- Anthropic can reject a resumed conversation if a tool-call message lacks its corresponding result, so systems must persist complete call-result histories.
- Tool outputs can contain malformed JSON, missing required fields, incorrect types, or invalid values such as an unsupported temperature unit.
- JSON Schema expresses machine-checkable requirements for syntax, shape, types, and allowed values.
- XGrammar supports grammar-based constrained decoding: invalid next-token logits are set to negative infinity, leaving probability only on tokens allowed by the grammar.
- Constrained decoding cannot ensure completion if the generation token limit is reached before the output reaches a valid final state.
- A Python tool can be exposed as a REST endpoint so it can be called remotely, rather than only from a local programmatic tool environment.
- FastAPI can turn Python functions into web endpoints and provide a JSON Schema that an agent can use to understand permitted calls.
- API keys can restrict access to endpoints and help protect personal data.
- A coding agent with bash access can also call REST endpoints directly using curl.
- Anthropic introduced the Model Context Protocol, or MCP, as a standard way for AI applications to discover and connect to tools and other context.
- FastMCP provides a way to build MCP servers similar to how FastAPI simplifies REST services, and REST API specifications can be adapted into MCP servers.
- MCP can keep an upstream credential, such as a GitHub token, on the server while giving the agent a separate MCP credential with more limited exposure.
- Compared with direct REST integration, MCP provides tool discovery and its own transports, and can also expose resources, prompts, and extensions.
- An agent can issue multiple tool calls in one response when the calls are independent, such as checking Pittsburgh weather, a calendar, and flight status.
- Dependent operations, such as finding a customer ID before fetching orders and refunding one, must remain sequential.
- Python async tools such as asyncio.gather can execute independent calls concurrently.
- Neubig links improved parallel calling in recent models to reinforcement learning incentives that reward finishing tasks quickly.
- The Berkeley Function Calling Leaderboard evolved from single-turn tests to include multi-turn tool calling, agentic tool use, and robustness.
- Useful measurements include selecting the right tool, supplying correct arguments, calling tools in the right order, and completing the task successfully.
- Evaluation should also track efficiency, such as token count and cost, along with timeout recovery, retries, and resistance to unsafe requests.
- Tool-call performance can vary sharply by inference provider: the cited OpenRouter comparison for the same model ranged from about 15% errors to roughly 0.01% to 0.05%.
- Serving differences can affect tool reliability, including inference algorithms, quantization, and whether grammar-based constrained decoding is enabled.
- Neubig reports that FP8 deployments generally produce fewer mistaken tool calls than heavily compressed FP4 deployments.
- Lossy speculative decoding may change outputs, while provider outages can also cause apparent tool-call failures.
- The tool-use stack combines schemas, model-generated calls, validation, dispatch, call-result IDs, concurrency, and evaluation; valid syntax still does not guarantee semantically correct code.
Transcript excerpt
0:00 Okay. So, hi everyone. Uh, welcome to class number two. Today, uh, today I'll be talking about tool use for language model agents. And, uh, this is, you know, the the most fundamental thing about agents that, uh, you know, makes them different from a language model. Um so the basic idea here is uh that we're going to be um a tool is basically an interface through which a language model can invoke an external computer program. And uh this is a language model cannot do this on its own because it's fundamentally a token predicting machine. But this enables you to do you know many many different things. So you
0:48 know you can search, you can call a calculator, you can call a browser, um you can call Python, you can call arbitrary APIs and so uh the language model kind of as you'll see the method for tool calling is standardized but what you do downstream with the tool call can be very different. So there's basically two uh benefits of tools and all of this is from one of the readings that I I posted on the website if you want to go back and read more about it. Uh this is a survey that uh I we wrote uh together Daniel and I and Zoro Wong, one of our our students. Um and the first thing is extending the ability of language models. So this is giving language models the ability to do
1:35 something that they fundamentally cannot do. And so this could be like accessing information or actions that are outside the language model's um you know action space. The second thing is uh facilitating the language model. And so what I mean by this is this is something that fundamentally the language model
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Graham Neubig.