Reviewing Code Nobody Typed: A Working Agreement for Agent-Written PRs

Typing a change does not establish understanding, and generating it does not transfer ownership to the tool. When a coding agent produces a large diff, the author still needs to explain its behavior, assumptions and verification.

There’s also evidence that people misjudge how much these tools help. In METR’s 2025 randomized study, experienced open-source developers took about 19% longer on tasks when allowed to use AI tools, while believing afterwards that they’d been about 20% faster. That’s one study, of one population, with the tools and tasks available at that time. METR’s February 2026 follow-up reports that selection and time-measurement problems make its newer experiment unreliable as an estimate of current productivity. Use the historical result as a reason to measure your own workflow, not as a claim that current tools slow every team.

What follows is a working agreement a team can adopt as it stands, and then adjust.

1. The PR says how it was made

Add three fields to the PR template:

## What I asked for
<!-- The task as given to the agent, or a faithful summary. Link the issue. -->

## What I verified myself
<!-- Specific things, not "tested it". E.g. "ran migration against a prod snapshot;
     row counts match", "read every IAM statement", "reproduced the bug first". -->

## What I did not check
<!-- Honest gaps. "Didn't test on Safari." "Didn't read the generated fixtures." -->

“What I asked for” lets the reviewer judge whether the change does the right thing, not only whether it’s internally consistent. Agents are very good at producing a correct solution to a slightly different problem. “What I did not check” is the most useful field. It tells the reviewer where to look, and it makes gaps a normal thing to admit rather than a thing to hide.

2. Whoever merges owns it

State it plainly, because the opposite belief spreads quietly: the person who merges a change owns it in production, exactly as if they’d typed every line. “The agent wrote that part” is never an explanation in an incident review.

The test is the one that has always applied: could the author explain this change, and debug it at 2am, without the agent? If not, the PR isn’t ready, however well it tests.

3. Review attention goes where agents are weakest

Agent-written code tends to be locally tidy: consistent naming, tests included, few typos. Reviewing it line by line for style wastes attention. Spend review time on the places where agents lack context:

• Fit with the system. Does it follow the existing pattern, or quietly introduce a second way of doing the same thing? Agents often reimplement a helper that already exists three directories away.
• Boundaries and assumptions. What does the code assume about its inputs, its callers and its environment? What happens at the edges?
• Tests that test nothing. Agents sometimes write tests that mirror the implementation, mock the thing under test, or were adjusted until they passed. Read the assertions and ask whether they’d fail if the code were wrong.
• Anything security-relevant. IAM policies, auth checks, input validation, secrets handling and network exposure. These are high-consequence areas where a plausible default is often the wrong one. An agent asked for “permission to launch tasks” will happily write a policy that permits far more (see the PassRole post).

4. Move mechanical checks from reviewers to CI

Repeated mechanical checks are candidates for automation; keep people focused on the decisions the checks cannot evaluate. Anything mechanical belongs in CI:

• formatting and lint, with no human comments about style;
• type checks, test runs and coverage thresholds on changed lines;
• policy checks on infrastructure plans, such as a Conftest rule that fails on unconditioned iam:PassRole (example);
• dependency and secret scanning;
• diff size: a PR over a set number of lines needs a stated reason, because giant agent-generated PRs are where review quality collapses.

The goal is to make routine failures visible before review while keeping test coverage and skipped checks open to scrutiny. The refund retry exercise gives reviewers a concrete example of a successful response hiding an incorrect effect count.

5. Keep the skill that review depends on

Reviewing well depends on being able to write and debug the code yourself. If nobody on the team has recently debugged the payment path by hand, nobody can review an agent’s change to it with confidence. Two cheap habits help:

• Rotate people through incident response and debugging work. Those tasks build understanding of the system that no amount of reading diffs does.
• In review, make “walk me through how this works” a normal question, not an accusation. It’s how the team finds out whether understanding is keeping pace with output.

Where this stops

A working agreement sets expectations. It doesn’t enforce them, and the PR template will decay into boilerplate if nobody reads it. Check every few months: pull ten merged PRs at random and see whether “what I verified” contains anything specific. If it doesn’t, talk about it with the team. Adding more template fields won’t fix it.