CMU AI Agents 2026: 7. Computer Use Agents
Overview
Jing Yu Koh explains how computer-use agents (CUAs) perceive screenshots, reason over interaction history, and operate graphical interfaces through actions such as clicks, typing, and scrolling. He traces benchmarks from World of Bits and WebArena to long-horizon OSWorld 2, then outlines a training pipeline combining large-scale grounding data, supervised demonstrations, synthetic trajectories, and reinforcement learning in simulated environments.
Key takeaways
- A computer-use agent is a closed-loop vision-language system: it conditions on a task, screenshots, and action history, then executes structured GUI actions and observes the resulting state.
- Static benchmarks such as ScreenSpot-Pro are inexpensive and fast, but can wrongly penalize valid alternative actions; end-to-end benchmarks such as WebArena instead score task outcomes through environment-state checks.
- Real-web benchmarks such as WebVoyager are closer to deployment but depend on fallible LLM judges and can impose unwanted traffic on website owners, making simulation valuable for repeatable evaluation and training.
- Long-horizon tasks need intermediate rubric rewards as well as final outcomes: OSWorld 2 tasks take a median human 1.6 hours, and a single binary score would hide partial progress.
- A practical CUA training stack combines abundant coordinate-grounding pretraining, supervised human and synthetic trajectories, and reinforcement learning in sandboxes.
- Privileged interface metadata can help a teacher model generate stronger trajectories, which can then be distilled into a screenshot-based agent that is more broadly deployable.
Chapters
- CUAs operate through the same graphical interfaces as people, including web browsers, desktop applications, and mobile devices.
- A model receives a task and an observation, usually a screenshot, then emits an action such as a click, scroll, swipe, or typed input.
- The environment executes each action and returns a new screenshot; the agent repeats the loop until it finishes or exhausts its budget.
- A task such as adding a blue mug under $20 to a cart requires the agent to inspect results, open an item, and continue from updated observations.
- OpenAI's 2017 World of Bits used simplified synthetic websites and reinforcement learning, but early models struggled even with these toy tasks.
- WebShop introduced more realistic simulated shopping in 2022, followed by WebArena's realistic web apps and OSWorld's desktop environments in 2024.
- Later benchmarks used coding agents to generate entire applications and synthetic data, while CUAWorld explored specialized software such as astronomy and biology tools.
- Koh cites a GPT-6 Astra CAD demonstration as evidence of rapid progress, while noting that the agent remained slow and expensive despite strong benchmark results.
- Computer-use success is harder to verify than math or software engineering because there may be multiple valid ways to complete a task.
- Static evaluations score individual actions against reference labels, such as whether a predicted click falls inside a target box.
- ScreenSpot-Pro tests visual grounding across roughly 1,500 screenshots from different applications and operating systems.
- Mind2Web evaluates next-action prediction from offline screenshots, making evaluation cheap but marking valid alternative actions as incorrect.
- End-to-end evaluation runs an agent in an environment and checks the final state rather than requiring every intermediate action to match a reference.
- WebArena includes about 800 tasks across four simulated web applications, with handwritten programmatic verifiers for task outcomes.
- For a repository-creation task, a verifier checks whether the requested repository exists after the agent finishes.
- Outcome-based checks better measure task completion, but incomplete verifiers can produce false positives and encourage reward hacking.
- VisualWebArena extends WebArena with 910 tasks designed to require screenshot-based, multimodal interaction rather than relying only on HTML.
- LLM or vision-language-model judges can assess a task and its action-and-screenshot trajectory when writing programmatic verifiers is impractical.
- WebVoyager tested agents on 15 real websites with 600 tasks; because real-site state is difficult to query, its evaluations used model judges.
- Online Mind2Web covered 136 websites and 300 tasks, and reported about 85% agreement between its judge and human ratings.
- Real-web benchmarks are costly and disruptive to website owners, so researchers increasingly prefer simulated environments.
- Human review remains the evaluation gold standard, but reviewing multiple trajectories per task is too expensive for frequent benchmark runs.
- OSWorld provides a Linux virtual machine with 369 tasks across nine applications, including Chrome and LibreOffice.
- A spreadsheet task can require finding receipts in a folder, entering values in LibreOffice Calc, and saving the result before programmatic checks run.
- Windows Agent Arena applies a similar virtual-machine evaluation approach to Windows workflows, including common enterprise tasks.
- As shorter WebArena and OSWorld tasks became easier for frontier models, research shifted toward workflows that take people hours.
- Long-horizon benchmarks add rubric-based intermediate checks because a single final pass-or-fail reward is too sparse to show partial progress.
- CMU's Odyssey benchmark uses 200 web tasks with multiple criteria, such as finding a nonstop Pittsburgh-to-Los Angeles flight and meeting a specified driving-time window.
- OSWorld 2 contains 108 tasks that take a median human 1.6 hours; examples include producing a CAD support bracket from a reference drawing.
- Koh also describes CUAWorld-Long tasks requiring at least 500 model calls, with recent models reportedly approaching 90% performance.
- Benchmark rewards can tolerate unwanted side effects: an agent may add too many products to a cart yet still satisfy a basic outcome verifier.
- Odyssey rubrics are typically generated by an LLM and checked by people; rubric-based rewards can make reward hacking harder than a single outcome check.
- Teams may train on tasks resembling WebArena or OSWorld, but should keep the actual benchmark test set separate; CUAWorld also provides training data.
- Accessibility trees can supplement screenshots with structured descriptions, while screenshots offer a more transferable interface across operating systems.
- A CUA is generally a vision-language model that receives a natural-language goal and visual observations, then generates a structured action.
- The agent's context grows by interleaving screenshots, prior actions, and task text, following a reasoning-and-action loop often called ReAct.
- Models tokenize text and screenshots into embeddings; action formats vary, including coordinate outputs, tool calls, and XML-like structures.
- Screenshot inputs simplify deployment across desktop systems, while accessibility metadata can add useful information when available.
- After an action, environments often wait about three seconds before capturing the next screenshot; agents may wait again if a loading screen remains.
- Most CUAs build on general pretrained vision-language models rather than being isolated models dedicated only to computer use.
- The training pipeline combines pretraining for visual grounding, supervised fine-tuning on demonstrations, and a later reinforcement-learning stage.
- Koh notes that computer use must handle practical infrastructure edge cases that do not arise in the same way for math or coding agents.
- Pretraining adds large-scale grounding examples that teach models to locate interface elements and predict their coordinates from screenshots.
- Website scripts can extract element boxes at scale, creating noisy but abundant localization data for early training.
- Action-prediction examples can pair page elements or task descriptions with the coordinates of the intended interaction.
- Normalized coordinates, such as values from 0 to 1,000 across the screen, help actions transfer across display resolutions.
- Supervised fine-tuning teaches the model to imitate screenshot-action trajectories recorded from people performing computer tasks.
- MOMOWeb and AgentNet are cited as sources of human demonstrations; MOMOWeb collected about 36,000 human trajectories.
- Synthetic data can scale demonstrations: MOMOWeb used a model with accessibility-tree access to generate about 105,000 trajectories.
- A stronger teacher can use privileged metadata unavailable at deployment, with lower-quality traces filtered by people or another model.
- Normalized coordinates help a grounding model transfer between screen sizes and devices, such as a Mac and a tablet.
- Agents may choose efficient alternatives to clicking, including Bash calls or writing code when those tools are available.
- Synthetic trajectories can be generated with extra interface information and then distilled into models that act from screenshots alone.
- The action space depends on the environment and can mix visual clicks with other tools rather than requiring GUI interaction for every step.
- In reinforcement learning, an agent attempts tasks in a sandbox, receives rewards from verifiers or judges, and updates its policy to favor successful trajectories.
- Simulation avoids repeatedly performing costly or risky real-world actions, such as booking rides or placing purchases.
- Koh reports that reinforcement learning can raise success from roughly 50% after supervised fine-tuning to 60% or more, and can support longer tasks lacking human demonstrations.
- The lecture closes by identifying speed, personalization, infrastructure, user experience, and multi-agent systems as open areas for computer-use research.
Transcript excerpt
0:00 Okay, great. So, yeah, today I'm super excited to have JY Code as our guest lecturer. So, um JY is an expert on um on multimodal machine learning um in particular computer use agents. Uh so, JY um has experience both in academia here as a PhD student CMU uh working with uh me and Russ Solakutnav but um also has a lot of industry experience. Uh so he worked at Google research before starting the PhD and um also he led the computer use agent team at Meta's TBD lab for a year and a half. >> Yeah. And so kind of like laid the foundation for um the strong agents that they're releasing now. And yeah, he also
0:49 um built uh some of the most uh impactful benchmarks in this area and also did work on um search for agents. So he'll also be giving a lecture later on in the course and we've covered some of his work already on multi-agent computer use and looking forward to uh you know hearing about the foundations of KUAS today. Thanks. >> Cool. Uh thanks a lot Daniel for the kind introduction. Um so today I'm very happy to talk about something that I've uh personally been very excited about for for a couple years now. Um I think today it's like taken for granted that we have uh that computer use is a is a core capability in like many frontier models. But uh as we will see by uh kind of stepping a bit through the history of
1:36 computer use agents, this was not always the case. Uh I think for a long time like um it was not very clear whether we would want computer use agents or how we would build strong computer use agents. Um so I think computer use agents or kas
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Graham Neubig.