Save this video free

CMU AI Agents 2026: 7. Computer Use Agents

Graham Neubig · 1:14:26 · Watch on YouTube

Part of Carnegie Mellon University 11-768. See all 14 lectures and search them

These notes were made with YouTube Collector. Paste any YouTube link and get the same: a summary, chapters and a searchable transcript.

Overview

Jing Yu Koh explains how computer-use agents (CUAs) perceive screenshots, reason over interaction history, and operate graphical interfaces through actions such as clicks, typing, and scrolling. He traces benchmarks from World of Bits and WebArena to long-horizon OSWorld 2, then outlines a training pipeline combining large-scale grounding data, supervised demonstrations, synthetic trajectories, and reinforcement learning in simulated environments.

Key takeaways

Chapters

0:00 Computer-Use Agents as Screenshot-Driven Observe, Reason, Act Loops
6:19 From World of Bits to Realistic and Long-Horizon Benchmarks
12:30 Static Evaluation with ScreenSpot-Pro and Mind2Web
18:09 End-to-End WebArena Evaluation with Programmatic Verifiers
22:45 VisualWebArena, LLM Judges, and Real-Web Evaluation
29:44 Human Review and Desktop Benchmarks: OSWorld and Windows Agent Arena
33:42 Long-Horizon Evaluation with Odyssey and OSWorld 2
41:17 Benchmark Tradeoffs, Rubrics, and Alternative Interface Signals
49:07 Cua Architecture: Interleaved Screenshots, Actions, and Context
57:11 Computer-Use Infrastructure and Multi-Stage Training
57:51 Pretraining for Visual Grounding and Coordinate Prediction
1:04:16 Supervised Fine-Tuning from Human and Synthetic Trajectories
1:07:38 Coordinate Transfer, Efficient Tools, and Beyond-Screenshot Actions
1:10:12 Reinforcement Learning in Simulated Environments and Remaining Challenges

Keep these chapters and the full searchable transcript in your own library.

Transcript excerpt

0:00 Okay, great. So, yeah, today I'm super excited to have JY Code as our guest lecturer. So, um JY is an expert on um on multimodal machine learning um in particular computer use agents. Uh so, JY um has experience both in academia here as a PhD student CMU uh working with uh me and Russ Solakutnav but um also has a lot of industry experience. Uh so he worked at Google research before starting the PhD and um also he led the computer use agent team at Meta's TBD lab for a year and a half. >> Yeah. And so kind of like laid the foundation for um the strong agents that they're releasing now. And yeah, he also

0:49 um built uh some of the most uh impactful benchmarks in this area and also did work on um search for agents. So he'll also be giving a lecture later on in the course and we've covered some of his work already on multi-agent computer use and looking forward to uh you know hearing about the foundations of KUAS today. Thanks. >> Cool. Uh thanks a lot Daniel for the kind introduction. Um so today I'm very happy to talk about something that I've uh personally been very excited about for for a couple years now. Um I think today it's like taken for granted that we have uh that computer use is a is a core capability in like many frontier models. But uh as we will see by uh kind of stepping a bit through the history of

1:36 computer use agents, this was not always the case. Uh I think for a long time like um it was not very clear whether we would want computer use agents or how we would build strong computer use agents. Um so I think computer use agents or kas

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Graham Neubig.

Want the full transcript?

Save this video in YouTube Collector to get its complete searchable transcript, your own AI summaries, and a library that keeps every video you collect in one place.