Picking the right model

Overview

Lucas (Anthropic applied AI) argues that “picking the right model” should be driven by a repeatable, private eval tailored to your production tasks, not by public benchmarks or token pricing. He proposes a process grounded in three pillars (model quality, latency, cost), shows how to build task-based evals with graders that verify both final outcomes and intermediate steps, and lists eval failure modes like mistaking noise for signal, infra/tool failures, and “silent saturation.” He then explains how to shift the quality/cost/latency Pareto frontier using Claude dials (thinking, effort), prompt caching (e.g., ~1/10 input token cost for cached prefixes with 80 to 90% hit rates), and context engineering, concluding with a workshop that sweeps Haiku/Opus/Sonnet across thinking/effort levels using TAO bench customer-service scenarios.

Key takeaways

Public benchmarks like SWE-bench verified and web/research leaderboards are only directionally helpful because production workloads are more heterogeneous (e.g., tool-using agents that research niche SDK details), so private evals must mirror your task mix.
Model selection should optimize for “cost per successful outcome,” not “cost per token,” which requires tracking completion/pass criteria in your eval dataset alongside latency and infra reliability.
Agent evals must grade both end results and intermediate steps (e.g., correct database/tool queries) because success can be achieved via different reasoning/tool paths that affect reliability and cost.
Eval results can be misleading if you don’t control for variance/noise (rerun tasks), separate infra failures from model behavior (API/tool-call errors), and update evals post-launch using real production traces (“silent saturation”).
Claude “thinking” and “effort” provide practical knobs to tune the accuracy/cost/latency Pareto curve, and better models can be faster in practice by using fewer turns/output tokens to reach the same pass rate.
Prompt caching and context engineering can shift the frontier substantially: prompt caching can make cached-prefix inputs cost ~1/10 list price (when you can reach ~80 to 90% hit rates), and compressing/cleaning tool responses can reduce tokens and improve accuracy simultaneously.

Chapters

0:00 Why model choice isn’t solved by “hot takes”: build a decision eval

Lucas frames model launches (model cards, prompting guides, benchmarks, social “hot takes”) as noise, emphasizing the business question: does the new model improve your specific use case?
He proposes a repeatable yes/no selection process backed by private evals rather than defaulting to “drop-in” model changes.
The simple baseline mapping (Opus for more intelligence, Haiku for lower latency/cost, Sonnet for balance) is shown as insufficient once you include reasoning/effort controls.

2:30 Three eval pillars: quality, latency, and cost, then “cheapest per outcome”

Lucas defines the three pillars for evals: model quality (task completion/accuracy), latency (especially for customer-facing flows), and cost.
He argues the optimal model is not “cheapest per token,” but “cheapest per successful outcome,” tying evaluation metrics to business success rates.
He highlights that public benchmarks (e.g., SWE-bench verified, web/research-style leaderboards) are directionally useful but rarely match heterogeneous production workloads like tool-using coding agents.

5:00 Design evals like exams: verify outcomes AND the steps taken

Lucas recommends building an eval dataset from atomic “tasks” with inputs, success criteria, and graders, analogous to showing your work on a math exam.
For agentic customer service, he suggests graders that check the final response and separate graders that verify correct tool/query behavior (e.g., SQL robustness via an LLM judge).
He also advocates deterministic checks for required tool calls and parameters (e.g., tool must be called; searches must be localized by customer country).

8:15 Common eval gotchas: noise-as-signal, infra failures, and silent saturation

Mistaking noise for signal: rerun every eval task multiple times and treat high metric variance as a sign the task/eval definition is misaligned or underspecified.
Infra failures: separate model quality from API/tool-call failures; score drops can come from tool/API problems rather than the model itself (e.g., Opus appearing worse due to call failures).
Silent saturation: keep eval data representative of production by collecting traces post-launch, capturing new failure modes, and feeding them back into the eval set.

12:30 Shift the Pareto frontier with Claude dials + prompt caching + context hygiene

Lucas explains two key model controls: adaptive “thinking” (scratchpad/system-two style deliberation) and “effort” (how much the model writes/uses tools across thinking, tool calls, and responses).
He gives counterintuitive examples from internal pipelines: enabling thinking/evaluating Haiku vs Sonnet vs Opus can reach the same pass rate with less time because better models may use fewer turns/tokens.
Frontier-shifting hacks: prompt caching can reduce cached prefix input cost to ~1/10 list price with typical 80 to 90% hit-rate targets; append-only messaging (avoid dynamic system prompt variables like time) helps preserve cache hits.

17:30 Tool-response compression and a workshop sweep: audit evals across models

Context hygiene: reduce tool response tokens by formatting (e.g., markdown instead of JSON), simplifying timestamps, and adding helpful fields like day-of-week to improve understanding and cut tokens (example claims include ~66.4% token reduction for a sports data tool).
He also cites deduplicating web search results before returning to Claude as another leverage point (example claims include ~77% input token reduction, ~65% cost reduction, and ~9% accuracy improvement).
Workshop: a TAO bench (airline customer-service slice) sweep audits evals across Haiku/Opus/Sonnet with thinking on/off and multiple effort levels, plotting pass rate vs output tokens and latency to choose an informed operating point.

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Claude.