CMU AI Agents 2026: 6. Agents for Coding and Software Development
Overview
Graham Neubig explains how coding models and software-development agents are built, from code-aware pretraining, infilling, and long-context training to tool use, evaluation, and reinforcement learning. He argues that reliable agents depend not only on model capability but also on high-quality tests, effective file-editing interfaces, realistic training environments, and support for the broader software-development lifecycle.
Key takeaways
- Code-model data preparation must preserve programming structure: whitespace, file boundaries, and code-to-documentation relationships matter, while duplicated forks and exposed credentials need filtering.
- Long-context coding performance depends on coherent repository examples, such as dependency-linked files, not merely increasing the context window; Qwen 2.5 Coder extended from 8K to 128K tokens with YaRN.
- Fill-in-the-middle training teaches models to use both left and right context, enabling code completion inside files, type inference, and variable renaming while retaining ordinary completion ability.
- Test-based evaluation accepts semantically different correct implementations, but weak or outdated tests and unstable dependencies can make benchmark scores misleading.
- Agent interfaces affect performance: dedicated file-editing tools and edit formats aligned with model training can be more reliable than forcing every change through generic Bash commands.
- Coding agents should be evaluated beyond isolated bug fixes: real software work includes planning, review, deployment, maintenance, test creation, and long-running changes across multiple issues.
Chapters
- Coding tasks range from one-off programs, such as visualizing a CSV, to modifying multi-file repositories and supporting broader software work.
- Graham Neubig focuses on coding-agent tools and model capabilities, building on earlier lectures about skills, memory, and long-context prompting.
- Single-step code generation, repository editing, and software development are related but distinct levels of capability.
- A code model must understand programming languages and preserve their syntax and conventions.
- Editing requires iterative improvement, which models need to learn rather than simply inheriting from one-shot code generation.
- Reasoning is especially useful in coding because program behavior can often be checked through execution or tests.
- Pretraining mixes source code with web text, technical explanations, documentation, and math so models retain general language and reasoning abilities.
- Code datasets require filtering autogenerated code, incorrect encodings, forks, vendored libraries, duplicates, and copied solutions.
- Data pipelines should track licenses, source dates, opt-outs, and sensitive information such as API keys; Neubig names TruffleHog as a credential-detection tool.
- Whitespace and file boundaries must be preserved, particularly because Python indentation carries meaning.
- Byte-pair encoding and other subword tokenizers need to represent code whitespace efficiently, including common indentation patterns.
- Tokenizing across whitespace can reduce token counts, but overly large tokens can impede autocomplete and partial-code generation.
- Training data and reinforcement-learning environments shape language preferences: models often default to Python or React and may need repeated instructions to produce Rust.
- Code Llama adapted Llama 2 with about 500 billion additional tokens, illustrating how continued training increases code exposure.
- Qwen 2.5 Coder experiments found that training on 100% code could hurt math and question answering; mixing text and math helped preserve general abilities.
- Long-context training should assemble coherent sequences, such as directory contents or files connected through dependency graphs, rather than blindly concatenating entire repositories.
- Qwen 2.5 Coder expanded context from 8K to 128K tokens using YaRN, with cross-file completion and infilling among the relevant evaluations.
- Fill-in-the-middle training lets a model generate code between a prefix and suffix, rather than only predicting text from left to right.
- Sentinel markers such as FIM tokens identify the missing span; multiple markers can represent multiple holes.
- Infilling supports tasks such as completing code within a file, inferring return types, generating comments, and renaming variables.
- Neubig describes results showing infilling improves middle-of-file generation without requiring a trade-off against ordinary code completion.
- Commit diffs paired with pull-request descriptions provide examples for learning how to edit existing code.
- Continuous-integration histories expose code versions that failed tests and later versions that passed, creating repair examples from public repositories.
- Execution traces can teach models to predict program outputs or stack states at specific points in a run.
- These data sources complement synthetic examples, which can be expensive to generate and may not reflect real development.
- BLEU-style metrics measure token overlap with a reference implementation, but can miss semantic differences or penalize equivalent code.
- CodeBLEU compares abstract syntax trees and data-flow relationships to go beyond surface-level token matching.
- CodeBERTScore uses contextual token embeddings to compare generated code with a reference implementation.
- Execution-based evaluation runs candidate code against tests, accepting different correct implementations but depending on test quality and stable environments.
- HumanEval evaluates generated Python functions against hidden unit tests, while CodeContests uses competitive-programming problems.
- Pass@1 measures whether one generated solution passes; Pass@k allows multiple attempts and counts success if any candidate passes.
- Execution benchmarks can have false positives from weak tests and false negatives when tests check requirements missing from the prompt.
- Code quality may also include runtime, latency, resource use, maintainability, and security, although early single-step training mostly rewarded correctness.
- A simple reinforcement-learning loop samples code, runs tests, and updates the model based on whether the implementation passes.
- Reasoning models are valuable for difficult single-step problems such as Codeforces tasks, where solutions must be correct and meet execution time limits.
- DeepCoder results illustrate continued gains from training across longer contexts, including stages at 16K, 32K, and 64K tokens.
- Readability can be included in post-training through human judgments and reward models, alongside functional correctness.
- A common repository workflow has three stages: localize the relevant code, edit it, and verify the change with tests.
- Agents repeat the cycle when verification fails, using error messages and failing tests to guide the next attempt.
- Agentless demonstrated an alternative: predict files, classes, functions, and lines in stages, then generate a patch without iterative tool use.
- That approach sometimes beat agents when models were better trained for single-step patch generation than for tool calling and verification.
- Mini-SWE-agent uses a single Bash tool for searching, editing, and running checks with commands such as ripgrep, find, sed, and pytest.
- Dedicated editing tools reduce the burden of choosing between shell commands and avoid fragile escaping, such as handling many backslashes in sed.
- Search-and-replace formats are often easier than line-numbered unified diffs, though repeated matching text can make edits ambiguous.
- The Aider code-edit benchmark showed that GPT-4 Turbo performed better with unified diff format, illustrating how model training and harness formats must align.
- Agents commonly locate code with ripgrep or other search tools; LocAgent explored tools for following dependency paths between related files.
- SWE-bench builds tasks from GitHub issues, repository states before the fixing pull request, and tests associated with the change.
- Its reduced test sets include fail-to-pass tests for the requested behavior and pass-to-pass tests that check for regressions.
- Running full test suites can be costly, so benchmark environments need dependable dependencies, commands, and focused tests.
- SWE-gym added training environments beyond SWE-bench evaluation tasks; Neubig reports that 500 examples improved a model from roughly 11% accuracy.
- SWE-rebench expands the pool of repository tasks using pull requests, with training and development sets for agent improvement.
- SWE-smith creates synthetic repair tasks by mutating working code and retaining tests that expose the introduced bug.
- Mutation-based tasks are lightweight and scalable, but can be less natural than real pull-request examples; multilingual benchmarks also need language-specific conventions.
- Models can perform poorly when an agent harness expects an edit format different from the one used in training, motivating training across systems such as OpenHands, OpenCode, and Codex.
- NVIDIA Nemotron reports training across multiple harnesses to improve compatibility with different coding-agent setups.
- SWE-bench Multimodal evaluates front-end issue fixes, where agents may need image understanding and browser-based verification.
- Browser checks can use visual inspection or Playwright scripts; Neubig notes that a non-multimodal model such as GLM-5.2 nevertheless performed strongly in a front-end design arena.
- The inner development loop covers editing and testing, while the outer loop includes code review, deployment, monitoring, maintenance, planning, and requirements.
- A 2019 Microsoft developer survey reported that coding took about 15% of developers' time, compared with 25% for meetings and email, 14% for debugging, and 8% for running tests.
- Other evaluation targets include app creation from scratch, backend-library reimplementation in ProgramBench, and maintaining code quality across multiple issues in a milestone.
- CodeScout targets localization, mutation-based benchmarks can evaluate test generation, and SWE-Playground and HybridGym explore automatically generated software-engineering tasks.
Transcript excerpt
0:00 Okay. Um, today I'll I'll be talking about agents for coding and software development. So, presumably many or mo most or all uh people here are using coding agents. So, um I I barely need to describe what their underlying functionality is. Um but I would like to talk about um how we can build coding agents and what the necessary things we need to have in order to build coding agents are. And um actually I just realized that the title of this slide is uh maybe uh a little bit of a misnomer. The first thing that I'd like to talk about is what it means to have a language model. Not a not an agent, but a language model that's able to write code. Um, and so
0:49 there's different uh things you can do. The kind of obvious thing is you say in English um I want you to write a program that uh visualizes this data for me. Um and you give it like a CSV uh file in your prompt and uh creates a bar chart uh for this data. Um, how many people do this sort of query like frequently? Doesn't necessarily need to be a bar chart, but like write write this program for me to do this task. Probably a lot of people uh and you write it in like chat GPT or a coding agent or something like that. Um, however, there's a lot of other tasks that involve generating code that's not just like a oneoff like prompt, but you
1:40 can also do um you can uh have it modify repositories. So, you can have it modify like multiple files. Um, and so this would be something more like in this repository uh fix this issue or make this improvement or something like that.
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Graham Neubig.