The Inference Gateway: One Place for Keys, Budgets, Failover and Logs
The first service that calls a model API holds its own API key, logs its own usage and handles its own retries. By the fifth service, you have five copies of that code, three places where keys live, no single view of spend, and no way to switch providers without five deployments. An inference gateway is a small service (or a managed product) that every model call passes through. It’s the same idea as an API gateway, applied to model traffic.
Open-source options include LiteLLM and Envoy AI Gateway, and cloud providers offer managed versions. Whether to build or buy matters less than deciding what the gateway is responsible for.
What belongs in the gateway
Credentials. Provider API keys live only in the gateway. Services authenticate to the gateway with their own identity: workload identity, IAM or mTLS, never a shared key. Rotating a provider key is then one change in one place, and a leaked service credential can be revoked without touching the others.
Budgets and rate limits per caller. The gateway knows which service, team or tenant made each call, so it can enforce limits per caller: tokens per minute, spend per day, a monthly allocation. Strict ceilings need atomic pre-call reservations and an allowance for delayed metering; checking yesterday’s billed total is not a hard cap. This is also where the provider’s shared rate limit gets divided fairly, so one batch job can’t starve interactive traffic. The rate-limit post covers the budgeting mechanics.
Uniform logging. One place emits a record of every call: caller, model, tokens, latency, cost, status. That is what makes a spend dashboard or a “which service is burning tokens” query possible. Keep full prompts and responses out of this log by default. They go to a separate, access-controlled store, if you keep them at all.
Retries and provider failover. Retry eligible transient errors with bounded backoff in one place. A timeout may follow a completed generation; account for duplicate provider cost, and do not replay downstream side effects when retrying the model call. If a provider or region is down, the gateway can route to another deployment of the same model: a different region, or the same model served through a different cloud. On Bedrock, cross-region inference profiles do some of this for you.
Model aliases. Services ask for reasoning-default or extract-fast, and the gateway maps each alias to a specific, dated model version. Upgrading a model becomes a gateway config change that you can roll out gradually and roll back, and that you run through evals first.
What doesn’t belong in the gateway
Prompt logic. Templates, few-shot examples and task-specific instructions belong to the services that own the task. A gateway that rewrites prompts turns into a second application nobody owns, and its changes affect every service at once.
Business rules about what the model may do. Deciding which tools an agent can call, or what needs approval, belongs to the application’s trusted execution layer, enforced by authorization and action validation. The gateway sees tokens, not intent.
Two features that change behavior without anyone noticing
Most gateway features are plumbing. These two change what your users get, so treat them as product decisions.
Failover to a different model
“If Model A is down, use Model B” sounds like reliability. In practice, the prompts, tool schemas and output parsing were tuned against Model A. Model B may follow the instructions differently, format output differently, or be more or less willing to call tools. An agent that fails over mid-session can behave oddly in ways that look like bugs in your code.
If you configure cross-model fallback:
- run your eval suite against the fallback model with the same prompts, and only list fallbacks that pass;
- record in the call log which model actually served each call, so incident reviews don’t assume the primary model;
- prefer failing over between sessions rather than within one, so a single conversation doesn’t switch models halfway through.
Semantic caching
Exact-match response caching still needs a validity boundary: include tenant, authorization scope, model/config version and relevant source-data version in the cache key, and define expiry. Temperature 0 does not guarantee deterministic output, and identical prompt text can depend on changed external state. Do not replay cached action-bearing responses as authorization to execute a tool.
Semantic caching (returning a stored response for a prompt that’s merely similar) is riskier than it sounds:
- “Cancel my order 1182” and “Cancel my order 1183” are very similar, and should never get the same response.
- A cache shared across tenants can return one customer’s answer, containing their data, to another.
- It’s almost never correct for agent steps, where the right next action depends on the exact state.
If you use it, scope the cache to a tenant, restrict it to specific read-only, non-personal endpoints (an FAQ answerer, for instance), and log cache hits so you can audit them. For agents, provider-side prompt caching of stable prefixes is the safer saving. It lowers the cost of re-sending the same system prompt and tools, and never returns a stale answer.
A minimal rollout
- Stand up the gateway with credentials, logging and model aliases only. Move one service onto it.
- Add per-caller budgets, set well above current usage, in alert-only mode for two weeks. Tighten using the data.
- Add same-model regional failover.
- Consider cross-model fallback and caching only after the evals exist to check them.
Where this stops
A gateway centralises control, which also means it centralises failure: if the gateway is down, every model call fails. Run it with at least the availability you expect from the providers behind it, keep it stateless so it scales horizontally, and keep it deliberately boring. It should be the least interesting service you operate.