Evals for Agents: Grade the Actions, Run It More Than Once, Decide Before You Look

An isolated unit test can often use fixed inputs and exact assertions. Agent evaluations frequently add model variability and several acceptable trajectories. The same input produces different trajectories from run to run, and “correct” is often a property of what the agent did, not a string it produced. Evals are how you get something like test discipline back. Most of the value comes from three decisions: what cases you test, how you grade them, and how you avoid fooling yourself with the results.

Cases: start from failures, not imagination

The best eval cases come from things that actually went wrong: incidents, escalations, bad outputs someone flagged. Each is a real input where you know the right behaviour. Add to them:

• Core tasks: the handful of jobs the agent exists to do, each with a clear success condition.
• Refusals: inputs where the right behaviour is not to act. The refund the customer isn’t owed. The instruction hidden in a document.
• Edge cases: empty results, tool errors, ambiguous requests.

Start with a manageable set tied to observed failures and core requirements; case count alone says little about coverage. Store each case with its input, any fixture state (database rows, files), and what counts as success.

Graders: check what the agent did

Grade in this order of preference:

1. End state. After the run, is the world in the right state? Is the ticket closed, the file changed correctly, the test suite passing? This is the most robust check, because it doesn’t care how the agent got there.
2. Action assertions. Properties of the trajectory: issue_refund was not called; verify_identity was called before update_address; fewer than 20 steps; no tool was called with an argument outside the allowed set. These stay stable across model versions in a way that assertions about wording don’t.
3. Code checks on outputs. The output parses, matches a schema, or passes a linter or policy check.
4. Model-graded rubrics. For qualities code can’t check (was the reply accurate and polite?), a grader model scores against a written rubric. Calibrate it: grade twenty cases by hand, compare with the grader, and fix the rubric until they agree. An uncalibrated LLM judge measures the judge.

Tools such as promptfoo and LangSmith’s trajectory evaluators cover much of this. Plain test frameworks with custom assertions over recorded traces work too.

Runs: once is an anecdote

Because agents are non-deterministic, a single run of each case tells you little. Run each case several times and report both:

• how often it passed (the pass rate across runs), and
• whether it passed every time. The τ-bench paper calls this pass^k. For an agent that takes real actions, “right 4 times out of 5” often isn’t acceptable, and the all-runs rate shows that where the average hides it.

Then the discipline that matters most: write down what result would change your decision before you run anything. “We ship the new prompt if pass^3 doesn’t drop on any refusal case and the core-task pass rate rises by at least 5 points.” Deciding after you’ve seen the numbers is how teams talk themselves into changes that don’t help.

A small worked comparison

Here’s a protocol for a question that sounds simple: does giving a coding agent a skill actually change what it writes? It uses the IAM skill linked at the end of this post, but the shape works for any instruction file or prompt change.

Task. One fixed prompt, taken from the skill’s before/after fixtures:

I have an orchestrator Lambda that needs to launch ECS worker tasks for my agents. Write the IAM role.

Conditions. The same tool, model and version, in a clean repository. Three runs without the skill installed and three with it. Record the tool version and model ID. The results only apply to that combination.

Grader. Review the rendered policy statements and effective permission assumptions against this fixed worksheet. Mark a row pass, fail or not evaluated, with the statement or missing artifact as evidence:

Check	Evidence required
Required launch path exists	Exact ecs:RunTask resource and cluster condition; missing permission is a failure, not a safe answer
PassRole is scoped	Required task and execution role ARNs, plus the expected iam:PassedToService value
Unexpected permission is absent	Inspect every Allow, action wildcard, NotAction, policy attachment and other relevant grant
Escalation controls match the requirement	Review the concrete Deny actions and conditions, trust policies, and any documented exception
Policy is complete enough to evaluate	Unknown or omitted statements are not evaluated, not a pass

A regex that finds a condition name anywhere in a file cannot establish this. In particular, all() over an empty list can mark a missing PassRole statement as scoped. Use a parsed policy and adversarial fixtures for an automated grader. The Conftest example demonstrates a limited structural check, with its own coverage gaps; it is not a full IAM evaluator.

Decision rule, written first. The skill “helps on this task” only if every with-skill run passes every worksheet check, with no not evaluated rows and at least two of the three without-skill runs fail one or more. Anything else is reported as inconclusive.

What it can’t tell you. Six runs of one prompt say something about that prompt, with that tool and model, on that day. They don’t show the skill improves IAM generally, or that the output is secure. Report unchanged and worse results alongside good ones. A comparison that only ever reports wins isn’t evidence.

For a separate action-regression exercise, When the Agent Acts Twice asks you to distinguish a successful response from the provider’s actual effects. It uses an offline synthetic refund rather than a cloud account.

Put it in CI

Once the suite exists, run it where it can block things:

• on every change to prompts, tool definitions or model configuration;
• against the exact bundle being deployed (prompt hash, tool schema hash, dated model ID);
• with results stored per run, so you can see trends and not just the latest number;
• with explicit, visible skips. “Evals skipped: no prompt changes” belongs in the job summary.

Model-graded evals cost money and time. Run the fast code-graded subset on every PR and the full suite before release.

Where this stops

Evals measure behaviour on the cases you wrote. Production will find cases you didn’t. That isn’t a failure of the eval suite, as long as every production failure becomes a new case. A suite that grows from incidents gets more useful every month. One written once and never touched becomes a ritual.