Token Spend: Attribute It, Cap It, Then Cut It

Model API spend tends to arrive as one line on one invoice. Behind it are many services, features and customers, plus agent sessions whose cost ranges from a cent to tens of dollars depending on how the loop went. Teams usually start by optimising: switching models, trimming prompts. It works better in this order:

1. Attribute every call to a caller, feature and outcome.
2. Cap spend per session and per caller, so no single bug can run away with the budget.
3. Cut cost where the attribution shows it’s concentrated.

1. Attribute every call

You can’t manage spend you can’t break down. Every model call should carry, at minimum:

• service and feature (support-agent, draft_reply);
• tenant or customer, if you serve more than one;
• session ID, so calls roll up into sessions;
• model ID and token counts: input, cached input, output.

Record this in the call log at the point where calls go out, ideally an inference gateway, so every service is covered without each one remembering. On Bedrock, application inference profiles can carry cost allocation tags, so spend appears by team or feature in Cost Explorer. For third-party APIs, your own logs are the only source of that breakdown.

Then turn cost into a unit metric tied to value: cost per resolved ticket, per document processed, per successful code change. “We spent $14,000 on tokens” can’t be acted on. “Cost per resolved ticket went from $0.38 to $0.61 after the 3 September release” can.

2. Cap spend where it can run away

Agent cost has a long tail. Most sessions are cheap, and a few loop, re-read huge tool outputs or retry forever. Averages hide that tail and invoices reveal it late. Caps enforced in code catch it as it happens:

• Per session: reserve estimated maximum call cost atomically before every model call and reconcile actual usage afterward. Include concurrent and already in-flight calls in the allowance. At the ceiling, stop cleanly and escalate. Don’t silently truncate and carry on.
• Per caller per day: service or tenant budgets enforced at the gateway, with an alert at 50% and 80%.
• Anomaly alerts: a session costing more than 10× the median for its feature, or a feature’s hourly spend doubling against the same hour last week.

Cloud billing and budget tools have reporting delay and may not expose per-session behavior. They’re a backstop. Session caps are the control.

3. Cut cost where it’s concentrated

With attribution in place, look at which features, and which steps within them, account for the spend. Then apply these levers in roughly this order, from least to most risky to quality:

Prompt caching. Agents re-send the same system prompt and tool definitions on every call. Caching that prefix is usually the biggest saving with no change in output. Anthropic charges a tenth of the input price for cache reads, and other providers offer similar discounts. To get cache hits, put the stable parts first and the variable parts last. Check the cached-token counts in responses to confirm it’s working.

Context compaction. Old tool results are re-sent on every later call. Replacing them with short stubs once they’ve been used cuts input tokens across the rest of the session (how).

Batch processing for anything not interactive: evals, backfills, nightly summaries. Batch APIs are typically half price in exchange for asynchronous delivery.

Output limits. Set max_tokens to what the step actually needs, and ask for structured output where possible. Output tokens usually cost several times as much as input tokens.

Model routing. Send simple steps (classification, extraction, routing itself) to smaller, cheaper models, and keep the large model for reasoning. This is the biggest lever and the riskiest. Run your evals on each routed step before and after, because a cheaper model that gets 10% of cases wrong can cost more in retries and escalations than it saves.

A worked session estimate

Illustrative numbers (use your provider’s current prices and your own traces):

• 12 model calls per session; system prompt and tools 6,000 tokens; context grows to about 30,000 by the last call
• average input ≈ 18,000 tokens per call → 216,000 input tokens per session
• 400 output tokens per call → 4,800 output tokens
• at $3 per million input and $15 per million output: $0.65 + $0.07 ≈ $0.72 per session

Now cache the 6,000-token prefix. On 11 of the 12 calls, 6,000 tokens are billed at a tenth of the price, which gives a gross cache-read saving of about 66,000 × $2.70 per million ≈ $0.18 per session (25%), before cache-write premiums or expiry misses. If you then reduce the average input size, recompute cached and uncached tokens separately; the two savings overlap. Context stubbing also changes the information the model receives, so re-run the evals.

Where this stops

Attribution and caps tell you where money goes and stop the worst surprises. They don’t tell you whether the spend is worth it. That’s the unit metric compared with the value of the outcome, which is a product decision. Sometimes the right move after all this is to spend more per session on a better model, because it resolves tickets that the cheap one escalates to people.