An Interview Exercise for the Agent Era: Review the PR the Agent Wrote

The classic coding interview measures whether someone can write a correct solution from scratch under time pressure. That’s still a useful signal, but it’s no longer most of the job. An engineer whose role includes reviewing coding-agent changes spends time on something else: deciding what to ask for, reading what came back, and judging whether it’s right, safe and consistent with the system. Few interview loops test that directly.

Here’s an exercise that does. Budget 45 minutes as a starting point; pilot the exercise before relying on that duration. Candidates can use whatever AI tools they’d use at work, because the exercise is about judgment, not recall.

The exercise

Give the candidate a pull request of about 150–250 lines, presented as written by a coding agent from a short task description. Include the task description. Ask them to review it as they would a teammate’s PR: comment, request changes or approve, and explain their reasoning out loud.

The PR is built so that most of it is fine, with several planted problems of different kinds. For example, for a team that runs agents on AWS:

Task given to the agent: “Add a Lambda that lets the orchestrator start ECS worker tasks for the summariser agent, with tests.”

The PR contains:

• A working Lambda handler with clean code and reasonable naming. Nothing to find here, and noticing that it’s fine is itself a signal.
• An IAM policy with iam:PassRole on "Resource": "*" and no iam:PassedToService condition. It deploys and works, and it may let the orchestrator run code as a more privileged role when ECS trust and the other required permissions permit it.
• A unit test that directly calls a mock of ecs.run_task and asserts that same mock was called, without invoking the handler. Removing the handler would leave this test green. A test that actually invokes the handler and checks the mock arguments can be useful.
• A new retry helper that duplicates one already in the codebase (include the existing one in the repo snapshot).
• A hardcoded cluster ARN with an account ID that doesn’t match the environment config.
• Not in the PR: handling for RunTask returning failures in its failures array. ECS can accept the call and still fail to place the task.

The IAM policy can come straight from real material: the “before” fixture in the free skill linked below is an agent-written policy with exactly this problem.

What you’re watching for

Score each area separately. A single overall impression is where bias gets in.

Area	Strong	Weak
Security judgment	Finds the PassRole issue and explains the escalation path in plain terms; proposes a scoped version	Misses it, or flags “wildcards are bad” without being able to say why this one matters
Test skepticism	Notices the test can’t fail; says what a meaningful test would assert	Counts the presence of tests as coverage
System fit	Checks for existing helpers; asks where the cluster ARN should come from	Reviews the diff as if the rest of the repo didn’t exist
Finding what’s absent	Asks what happens when a task fails to start	Only comments on lines that exist
Prioritisation	Separates blockers from nits; would block on security and the useless test, not on naming	Treats every comment as equally important
Communication	Comments a teammate could act on without a follow-up conversation	Vague (“this seems off”) or harsh

Nobody will find everything in 45 minutes. The ordering tells you more: which problems they found first, and which they’d block on.

Running it fairly

• Allow AI tools, and watch how they’re used. A candidate who pastes the diff into a model and reads out the answer is showing you something. So is one who uses a model to confirm a specific suspicion (“does RunTask require PassRole on the task role as well as the execution role?”). Both are realistic. The second is what you want.
• Give every candidate the same PR and the same prompts, and calibrate interviewers on a few recorded sessions before using it for decisions.
• Adapt the planted problems to the role. For a frontend role: an accessibility regression, a test that snapshots everything, a duplicated component. Keep the structure (mostly fine, a few problems of different kinds, one important omission).
• Rotate the PR once it’s likely to have leaked, and keep a few versions.

Pair it with a short build task

Reviewing is half the job. The other half is directing the agent well. A second 30-minute exercise: give the candidate a small, clearly specified task and a coding agent. Watch how they break down the request, what context they provide, how they check the result and what they do when it’s wrong. You’re looking for someone who verifies before they trust, not someone who types fastest.

Where this stops

This exercise measures review judgment on one kind of change in a controlled setting. It doesn’t measure how someone handles ambiguity over weeks, works with a team, or designs a system. Like any work sample, it’s evidence to weigh with the rest of the loop, not a replacement for it. It’s also only as good as its planted problems. If the problems don’t reflect the mistakes your team actually sees, you’ll be hiring for the wrong skill.