A Launch Checklist for Agents That Take Real Actions

Most “production readiness” lists for AI agents are production readiness lists for any service with the word “agent” added: health checks, autoscaling, dashboards. You need those. But they miss what makes agents different. An agent chooses its own actions, and the checklist should be organised around those actions.

This is the list I’d want finished before an agent with write access meets real users. It’s deliberately concrete: each item is either done or not.

1. Inventory every action

Write down every tool the agent can call, and for each one:

Tool	Side effect	Reversible?	Max impact per call	Guard
search_orders	none	n/a	n/a	none
update_address	edits a customer record	yes (audit log)	one record	schema validation
issue_refund	moves money	no	order total	approval above $50
send_email	external message	no	one customer	template only, no free text

This table is the single most useful launch artifact. Almost every later item is decided by it. Irreversible actions with large maximum impact need the most controls. Read-only tools need the fewest. If filling in “max impact per call” is hard, the tool’s interface is too broad. Narrow it before launch: issue_refund(order_id) capped at the order total, not create_payment(amount, destination).

2. Hard limits in code

Set these outside the model, where a confused or manipulated model can’t talk its way past them:

• maximum steps per session
• maximum tokens and maximum spend per session
• maximum wall-clock time per session
• maximum calls per tool per session (three refunds in one conversation is a signal, not a feature)
• maximum aggregate impact per period (total refunds per hour across all sessions)

The aggregate limit is the one people forget. It catches the failure where every individual session looks reasonable and the sum doesn’t.

3. Permissions scoped to the inventory

• The agent’s credentials allow exactly the tools in the table, and nothing else the underlying API offers.
• Credentials are per environment. The staging agent can’t reach production data.
• Roles that can launch or invoke other workloads have scoped iam:PassRole and an explicit escalation deny (details).
• Every role has a written blast-radius note: if the agent is fully compromised, what’s the worst it can do?

4. Approval gates for irreversible actions

• Every irreversible action above a threshold goes through a person.
• The approver sees the exact action and arguments, not the model’s summary of them.
• Approval requests expire. An approval given yesterday doesn’t authorise today’s call.
• Approvals are logged with who, when and what they were shown.

Keep thresholds high enough that approvers aren’t clicking through dozens a day. A gate people approve without reading is worse than none, because it moves responsibility without adding a check.

5. Idempotency and retries

• Every side-effecting tool preserves a durable operation ID and approved payload across retries; repeated requests return the same outcome without another effect, or stop for reconciliation.
• Retries happen at the tool-call level, not the session level.
• Test an external effect followed by a lost response, a restart, concurrent attempts and changed retry arguments. A timeout must remain unknown until reconciled. The offline retry lab supplies one synthetic example.

6. Kill switches

• One flag disables the agent entirely and routes users to the fallback path, without a deploy.
• One flag per irreversible tool disables that tool while leaving the rest working.
• Someone on call knows where the flags are and has used them in a drill.

A per-tool switch is what makes a Friday-evening incident manageable. You turn off issue_refund, the agent keeps answering questions, while the incident owner decides whether the remaining behavior is safe enough to keep running.

7. Tests on actions, not only answers

• A fixed set of test conversations, each with assertions on which tools were and weren’t called, and with what arguments.
• Adversarial cases: a user asking for a refund they aren’t owed, a document containing instructions, a tool result that looks like a command.
• These run in CI against the exact prompt, tool schema and model version being deployed.
• Every incident after launch adds a test case.

Assertions about actions are far more stable than assertions about text. “Must not call issue_refund” stays true across model versions. “Response should mention the policy” doesn’t.

8. Monitoring for the first week

Watch, daily at first:

• tool-call rate per tool, compared with what the inventory predicted;
• blocked calls: every time a guard, limit or approval rejection fired;
• sessions that hit a hard limit;
• cost per session, as a distribution rather than an average;
• escalations to humans, and the reasons.

Blocked calls deserve the most attention. Each one is the agent attempting something the design says it shouldn’t. A few are expected. A trend means the prompt, the tools or the users aren’t what you designed for.

9. Tell users what they’re talking to

• Users know they’re dealing with an automated agent, and how to reach a person.
• Check the transparency rules that apply to you. In the EU, the AI Act’s Article 50 transparency obligations for systems that interact with people apply from August 2026.

Where this stops

A checklist covers the failure modes you thought of in advance. It doesn’t make the agent’s judgment good. It helps constrain and observe consequences. Some permitted actions remain irreversible; name those explicitly in the release decision. That’s the realistic goal for a first launch, and it’s what earns you the evidence to loosen limits later.