Event-Driven Agents: Duplicates, Loops and Recovery After a Crash

Putting agents behind events instead of direct calls has real benefits. A triage agent publishes ticket.classified, and a research agent, a billing agent and an analytics pipeline can each subscribe without the triage agent knowing they exist. Slow agents don’t block fast ones, and a crashed consumer catches up from its queue.

Event-driven systems also have well-known failure modes. With ordinary services those are manageable. With agents they get expensive, because each event handler is a sequence of model calls costing real money and possibly taking real actions. Two matter most: duplicates and loops.

Duplicates: at-least-once means sometimes twice

Amazon EventBridge and SQS standard queues deliver at least once. EventBridge retries failed deliveries to a target for up to 24 hours by default. SQS redelivers any message whose visibility timeout expires before it’s deleted. Your handler will see some events twice.

For a stateless transform that’s harmless. For an agent that spends two dollars researching a ticket and then posts a reply to the customer, a duplicate means double the cost and two replies.

The defence has two layers:

1. Track work durably, including incomplete work. Every event carries a stable event_id. The following conditional write demonstrates exclusive admission only. It is deliberately incomplete: a crash after claiming can leave unfinished work looking like a duplicate. Do not use it as a production handler:

import time

import boto3
from botocore.exceptions import ClientError

table = boto3.resource("dynamodb").Table("processed-events")

def claim(event_id: str, handler: str) -> bool:
    try:
        table.put_item(
            Item={"pk": f"{handler}#{event_id}", "ttl": int(time.time()) + 7 * 86400},
            ConditionExpression="attribute_not_exists(pk)",
        )
        return True
    except ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return False
        raise

2. Recover incomplete work and deduplicate the effect. Use explicit pending, in-progress, unknown and completed states, with a lease/fencing strategy appropriate to the store. A retry may acknowledge a completed operation, but an incomplete claim needs recovery. Persist a stable logical operation ID and its approved payload before the external call; reuse both across retries and process restarts. A response timeout leaves the external outcome unknown. If the provider cannot deduplicate or authoritatively reconcile that operation, stop for reconciliation rather than issue a new action. Writing a completion marker after an external effect leaves a crash gap; the two are not atomic simply because the marker lives beside the claim.

The refund retry exercise demonstrates this gap locally with synthetic data. It also shows why a completed-effect count is stronger evidence than a successful client response.

If you need strict ordering per entity (all events for one ticket, in order), SQS FIFO queues give you ordering within a message group and deduplication within a five-minute window. Five minutes is shorter than many agent sessions, so FIFO deduplication doesn’t replace the handler-level claim.

Loops: agents that trigger each other

This is the failure that’s more specific to agents. The research agent finishes and publishes ticket.updated. The triage agent subscribes to ticket.updated, re-classifies, and publishes ticket.classified. That triggers research again. Each hop is reasonable on its own, and together they’re an infinite loop that makes model calls on every cycle.

Ordinary services can form event loops too. Agents add a source of variability when model decisions influence which events are emitted. A prompt change, or the model choosing to “update the ticket with its findings”, creates a new edge in the graph that nobody drew.

Three guards, in increasing strength:

• Don’t react to your own output. Every event records its source_agent. Handlers ignore events whose causation chain already includes themselves. That’s cheap, and it catches direct ping-pong.
• Carry a hop count and stop at a limit. Each event records how many agent handlers are upstream of it. A handler that receives hops >= MAX_HOPS refuses to publish anything further and raises an alert. This catches loops of any length.
• Budget by correlation ID. Everything that descends from one originating event shares a correlation_id. Track the total spend per correlation ID, and stop all handlers for it at a ceiling. That catches runaway fan-out as well as loops.

The envelope

All of the above depends on metadata that every event carries. Put it in a standard envelope and reject events without it:

{
  "event_id": "evt_01JA3Y6M9Q...",
  "type": "ticket.classified",
  "occurred_at": "2026-07-29T14:02:11Z",
  "correlation_id": "cor_01JA3Y5ZKT...",
  "causation_id": "evt_01JA3Y5ZKV...",
  "source_agent": "triage@2026-07-20.2",
  "hops": 2,
  "data": {
    "ticket_id": "T-88121",
    "label": "billing_dispute",
    "confidence": 0.91
  }
}

correlation_id ties everything from one user request together for tracing and budgets. causation_id points at the exact event that caused this one, so you can rebuild the chain. source_agent includes the agent’s version, because “which version of triage emitted this?” is the first question after a bad release.

This pseudocode locates the checks; claim must be replaced with a recoverable state machine, not the incomplete admission example above. Concurrent budget updates require atomic reservation:

MAX_HOPS = 4

def handle(envelope, handler_name, work):
    if handler_name in causation_chain(envelope):     # looked up by causation_id
        return skip("self-trigger")
    if envelope["hops"] >= MAX_HOPS:
        alert("hop limit", envelope)
        return skip("hop limit")
    if spend_for(envelope["correlation_id"]) >= CORRELATION_BUDGET:
        return skip("budget")
    if not claim(envelope["event_id"], handler_name):
        return skip("duplicate")
    return work(envelope)   # any events it publishes get hops + 1

Dead-letter queues and replay

Configure a DLQ on every EventBridge target and SQS queue. For agents, a message in a DLQ is often not a transient failure. It may be a prompt the model can’t handle, or an input that triggers a guard. Look at DLQ messages before replaying them. Replay deliberately with the original operation identity. Confirm that claim state permits recovery and that the provider still retains the deduplication record; otherwise a replay may be silently skipped or repeat an effect.

Where this stops

Events decouple the agents’ code, not their behaviour. When one agent’s output format drifts after a model upgrade, every subscriber is affected, and nothing in the event bus will tell you. Version your event schemas, validate data against them in each handler, and treat a schema validation failure as an alert, not a log line.