Do You Need a GPU for Your Agent? A Break-Even Calculation

If your agent calls a hosted model API (Anthropic, OpenAI, Bedrock, Vertex), the GPUs are someone else’s problem, and the right number of GPUs for you to run is zero. Your agent’s own process spends its time waiting on HTTP. It needs a small container, not an accelerator.

A GPU enters the picture only when you decide to run a model yourself. There are real reasons to: data that can’t leave your environment, a small fine-tuned model for a narrow task, predictable costs at high volume, or latency requirements an API can’t meet. This post covers how to size that decision.

Step 1: will the model fit?

GPU memory has to hold the model weights, plus the KV cache for every request in flight, plus runtime overhead.

Weights: parameters × bytes per parameter.

KV cache: grows with context length × concurrent requests. For agents this is the part that surprises people, because agent contexts are long: tool outputs accumulate. A model that fits comfortably with one short request can run out of memory with eight concurrent 30,000-token agent sessions. Inference servers such as vLLM manage this with paged allocation, but they can’t create memory. Leave room for the KV cache, not just the weights.

A rough fit guide for common cloud GPUs:

• 24 GB (NVIDIA L4, A10G): an 8B model at FP16 with modest context and concurrency, or a quantised model of up to roughly 14B with room to spare.
• 48 GB (L40S): an 8B model with long context and real concurrency, or a quantised ~30B model.
• 80 GB (H100): a quantised 70B model on one GPU. At full precision, 70B needs several.

Step 2: what throughput will you actually get?

Throughput on GPUs depends heavily on batching. An inference server with continuous batching serves many requests at once, and total tokens per second rises steeply with concurrency. Single-request speed is a misleading benchmark for an agent fleet. Benchmark your own model, at your own context lengths, at your expected concurrency, with the server you’ll deploy. vLLM ships a benchmarking tool for exactly this.

Measure two numbers:

• sustained output tokens per second across all concurrent requests at the latency you can accept;
• time to first token at that load, because agents make many sequential calls and latency compounds across a session.

Step 3: the break-even

With a measured throughput, the self-hosted cost per million output tokens is:

cost per 1M tokens = hourly_cost / (tokens_per_second × 3600 × utilization) × 1,000,000

Utilization is the term that decides everything. It’s the fraction of paid GPU hours spent doing useful work. A GPU you pay for around the clock that’s busy four hours a day is at about 17% utilization.

A worked example with illustrative numbers (substitute your own measurements and current prices):

• instance at $1.00/hour
• measured 1,500 output tokens/second at acceptable latency
• at 100% utilization: 1.00 / (1,500 × 3,600) × 1,000,000 ≈ $0.19 per million tokens
• at 20% utilization: ≈ $0.93 per million tokens
• at 5% utilization: ≈ $3.70 per million tokens

Now compare with the complete API cost for a model of similar quality on your task, including input tokens, output tokens, cache charges and failed work. The GPU throughput must be measured with that same input/output mix; the output-only arithmetic above is not a complete API break-even. That last part matters more than the arithmetic. If the small self-hosted model only matches the API model on 80% of your cases, the 20% it gets wrong have a cost (retries, escalations, bad actions) that belongs in the comparison.

Then add what the calculation leaves out:

• engineering time for deployment, upgrades, monitoring, and on-call for a new kind of service;
• capacity for peaks and redundancy, reflected in paid GPU count and utilization; avoid adding idle cost again if the utilization term already includes it;
• model upgrades: a new open model means re-benchmarking, re-evaluating and redeploying, whereas an API upgrade is a config change and an eval run.

When self-hosting usually wins

• High, steady volume on a narrow task. Classification, extraction or routing at millions of calls a day, where a small model fine-tuned on your data matches a large general one. That’s the case in this post on small models.
• Hard data-residency requirements that no available hosted offering in your region meets.
• Batch workloads you can schedule to keep utilization high, then shut the GPUs down.

When it usually loses

• The agent’s main reasoning model. Frontier-quality reasoning on long, tool-heavy contexts is expensive to reproduce, and the hosted models improve every few months without any work from you.
• Spiky, low-volume traffic, where utilization stays in single digits.
• Teams without someone who already wants to operate GPU inference.

Where this stops

This calculation compares running costs. It doesn’t tell you whether the self-hosted model is good enough. Only an evaluation on your own cases can. Run that first. If the smaller model can’t do the task, the break-even point is irrelevant.