Your Agent Scales Until the Rate Limit: Queues, Backpressure and Token Budgets

Scaling a web service usually means adding compute. Scaling an agent workload usually means discovering that compute was never the constraint. The bottleneck is the model provider’s rate limit, a quota you share with every other workload in the account. What happens when you hit it depends mostly on decisions you made before you hit it.

Know which limit you’ll hit first

Model APIs limit more than one thing. Anthropic’s API, for example, limits requests per minute, input tokens per minute and output tokens per minute separately, per model. Amazon Bedrock sets per-model, per-region quotas on requests and tokens per minute. Agents are unusual because each loop iteration re-sends the growing conversation, so input tokens per minute is typically the first limit reached. A single long-running session can use a large share of it.

Work out your ceiling before you need it:

sustainable sessions/min ≈ ITPM limit / (avg input tokens per call × avg calls per session)

If each session makes 12 calls averaging 20,000 input tokens, a 2,000,000-ITPM limit supports about 8 sessions a minute before anything else goes wrong. Measure the two averages from your traces rather than estimating them. They’re usually larger than people expect, because context grows through the session.

Put a queue in front of the agent

A synchronous agent endpoint turns a traffic spike directly into a rate-limit spike. A queue turns it into a backlog. On AWS the standard shape is:

API -> SQS queue -> Lambda or ECS consumer (capped concurrency) -> model API
                  \-> DLQ after N failed receives

Three settings do most of the work:

• Consumer concurrency cap. For Lambda, set maximum concurrency on the SQS event source mapping. This caps how many sessions can call the model at once, whatever the queue depth. Derive it from the ceiling calculation above, not from Lambda’s defaults.
• Visibility timeout longer than the longest session. If a session can run 10 minutes, a 5-minute visibility timeout means SQS hands the same message to a second consumer while the first is still working. You get two agents doing the same job and double the spend.
• A dead-letter queue with a small maxReceiveCount. Choose maxReceiveCount from your retry and outage expectations; three is an example, not a diagnostic rule. Park it where a person can look at it.

The queue changes your user-facing contract from “respond now” to “accepted, result later”. For many agent tasks (reports, triage, research, code changes) that’s the honest contract anyway.

Retries: respect the server, add jitter, stop eventually

A 429 means the provider wants you to slow down, and it usually says for how long: honour the retry-after header when it’s present. Otherwise use exponential backoff with full jitter, so a hundred consumers throttled at the same moment don’t all retry at the same moment:

This retry fragment assumes a numeric, validated retry_after value and a client-specific RateLimited exception; cap total elapsed time as well as attempt count.

import random, time

def call_with_backoff(fn, max_attempts=6, base=1.0, cap=60.0):
    for attempt in range(max_attempts):
        try:
            return fn()
        except RateLimited as e:          # your client's 429 exception
            if attempt == max_attempts - 1:
                raise
            delay = e.retry_after or random.uniform(0, min(cap, base * 2 ** attempt))
            time.sleep(delay)

Two agent-specific points:

• Retry the model call, not the session. If call 9 of 12 is throttled, retry call 9. Restarting the session repeats eight calls you already paid for and may repeat their side effects.
• Side effects must be idempotent before you add retries. If a tool call creates a ticket or sends an email and the next step fails, a naive retry does it again. Give each side-effecting tool call a durable logical-operation key with an immutable approved payload. A session/step key works only if it survives restart and always denotes the same action. Have retries retrieve the original result without another effect, or reconcile an unknown outcome. On Lambda, Powertools’ idempotency utility tracks idempotency state with a persistence store. It cannot by itself make a separate external effect atomic with your completion record; check the downstream contract and crash recovery.

Share one token budget across workers

Concurrency caps limit sessions, but sessions vary a lot in size. If several services share one provider account, they also compete for the same limit without knowing about each other. A shared token bucket fixes both: before each call, a worker reserves its estimated input tokens from a central budget (a Redis or DynamoDB counter refilled each minute), and waits if the bucket is empty.

That turns the provider’s hard limit into a soft internal one you control. It also gives you somewhere to put priorities. Interactive sessions can draw from the bucket before batch jobs, and a runaway session can be cut off at its own ceiling before it drains everyone else’s budget.

Reduce tokens per session before you ask for more quota

Quota increases take time, and every token you don’t send is one that doesn’t count against the limit:

• Prompt caching. Stable prefixes (the system prompt and tool definitions) can be cached. Anthropic bills cache reads at a tenth of the normal input price, and some providers’ rate limits treat cached tokens differently from uncached ones. Check how yours does before counting on the headroom.
• Compaction. Summarise or drop old tool outputs once they’ve been used. A 40,000-token log from step 2 rarely needs to be re-sent at step 11.
• Routing. Send simple steps (classification, extraction) to a smaller model with its own, separate limit.
• Batch APIs for anything that isn’t interactive. They are typically half price and run against separate capacity.

Where this stops

Queues and budgets make overload graceful. They don’t create capacity. If sustained demand is above your limit, the backlog grows without bound and results arrive later and later. Alert on queue age (the oldest message’s age), not only queue depth, and treat a steadily growing age as a capacity problem that needs a quota increase, a cheaper model, or fewer tokens per session.