Event-Driven Architecture for Multi-Agent Systems

The first multi-agent system I built had every agent calling every other agent directly, function-call style, because that’s the natural extension of how you’d call one agent. It worked for a demo with three agents and fell over almost immediately in anything resembling production — one slow agent stalled the whole chain, one agent’s malformed output silently corrupted the next agent’s input, and adding a fourth agent meant touching the call sites in all three existing ones. That’s the same problem service-to-service architectures solved by moving from direct calls to events, and the solution transfers, with a few agent-specific wrinkles.

Choreography vs Orchestration: Pick Deliberately, Not by Default

There are two broad shapes for coordinating multiple agents, and the choice isn’t cosmetic.

Orchestration means a central coordinator (which may itself be an agent, or may just be a workflow engine) decides what happens next and dispatches work to other agents. It’s easier to reason about — there’s one place to look to understand the flow — and easier to add global policies to, like escalation rules or budget limits. The cost is a bottleneck: the orchestrator has to understand every agent it coordinates, and it becomes the thing every new capability has to be threaded through.

Choreography means agents react to events and publish events of their own, with no central coordinator deciding the sequence. Adding a new agent that reacts to an existing event type requires no changes to the agents already publishing it. The cost is that understanding “what actually happens when X occurs” requires tracing event flow across the whole system rather than reading one orchestrator.

I built Agora, a message router for coordinating multiple agents, around a middle path: choreography for how agents discover and react to work, with a thin orchestration layer for the handful of flows — budget enforcement, escalation, overall task status — that genuinely need a single source of truth. Pure choreography without any central oversight tends to make “is this task actually done” a surprisingly hard question to answer; pure orchestration tends to make the coordinator an ever-growing bottleneck. Most systems land somewhere in between, and the right split is worth deciding on purpose rather than defaulting into.

Message Contracts an Agent Can’t Misinterpret

Agents are worse at silently tolerating ambiguous input than typical services are, because an LLM will often produce a plausible-looking response to a malformed message instead of throwing a clean error. That makes strict message contracts more important here than in most distributed systems, not less.

A few rules I hold to for every event an agent produces or consumes:

• Explicit schema, validated at the boundary. Every event has a versioned JSON schema, and both producers and consumers validate against it before doing anything else. Don’t let an agent “interpret” a malformed event — reject it before it reaches the model.
• No implicit fields. If an event’s meaning depends on a field being absent versus present versus null, that ambiguity will eventually get exploited by a model that fills in something reasonable-sounding but wrong. Make every required field required, and every optional field’s absence mean exactly one thing.
• Include provenance, not just payload. Every event should carry which agent (and which version of it) produced it, and what upstream event triggered it. When something misbehaves three hops downstream, that’s how you trace it back.

{
  "eventType": "research.summary.completed",
  "schemaVersion": "2",
  "taskId": "task_8f2c",
  "producedBy": { "agent": "research-agent", "version": "1.4.0" },
  "causedBy": "research.query.dispatched:evt_991",
  "payload": {
    "summary": "...",
    "sourcesConsulted": 6,
    "confidence": "high"
  }
}

The schemaVersion and causedBy fields aren’t decoration — they’re what let you evolve one agent’s output format without breaking every consumer at once, and what let you actually debug a multi-agent chain after the fact.

Idempotent Consumers and Dead Letters for Flaky Output

Agents fail in a specific way traditional services usually don’t: they can produce output that’s syntactically valid and semantically wrong — a summary that misreads the source, a classification that’s confidently incorrect. Your event infrastructure can’t fix that, but it can stop it from cascading.

• Make every consumer idempotent against redelivery. Message brokers deliver at-least-once; if an agent consumes the same event twice because of a retry, it should produce the same result, not double the side effects. Dedupe on event ID before acting.
• Validate output before publishing it downstream, not just input before consuming it. An agent that emits an event failing its own schema, or with an out-of-range confidence score, should route to a dead-letter queue rather than propagate.
• Give dead-lettered messages a second life path, not just a graveyard. Route them to a human review queue or a retry-with-different-agent path — the point of catching a bad output is to do something with it, not just to stop it from spreading.

Treating agent output as untrusted input to the next stage — the same way you’d treat data from an external API — is the single biggest mindset shift that makes multi-agent event systems reliable instead of fragile.

Key Takeaways

• Choose choreography or orchestration deliberately per flow rather than defaulting to one architecture everywhere — orchestration for anything needing a global source of truth, choreography for everything else.
• Define strict, versioned schemas for every inter-agent event, eliminate implicit/ambiguous fields, and carry provenance (producer, version, causing event) on every message.
• Make every agent consumer idempotent against at-least-once delivery, and validate an agent’s own output against schema before it’s allowed downstream.
• Route invalid or low-confidence agent output to a dead-letter path with a real second life — human review or reprocessing — not a silent graveyard.