Ask what inference costs and you will be quoted a rate per GPU-hour because that is the number infrastructure providers publish. It is close to useless for a business case.
Identical hardware serving an identical model can produce costs differing by more than an order of magnitude and the variable driving most of that spread sits with the operator rather than the vendor.
The unit decides the answer
A GPU-hour rate prices the input. What a product needs priced is the output: cost per million tokens for a language workload, or cost per completed transaction for anything else.
That is the marginal cost of one more customer action, and it is the only figure that survives contact with a finance team.
Throughput separates the two, and throughput belongs to the deployment rather than the accelerator. The same hourly rate therefore resolves to wildly different unit costs depending on how the workload is served.
Utilisation is the dominant variable
A costing study, Beyond Per-Token Pricing, swept the offered request rate across 42 benchmark runs on H100 hardware and measured what that does to effective cost. For a dense Llama 3.1 8B deployment on a single GPU billed at $6.98 per hour, cost per million output tokens ran as follows:
- One request per second: $7.60. The GPU is arrival-limited at a batch size of one, and the rental is amortised over a trickle of tokens.
- Ten requests per second: $0.79. An arrival rate few teams would describe as low, still carrying a 2.5-fold penalty against the saturated figure.
- Twenty-five requests per second: $0.37. Within 20% of the floor.
- Fifty requests per second: $0.32. Effectively the hardware ceiling.
The mechanism is Little’s Law. Offered request rate sets in-flight concurrency, concurrency sets achievable batch size, and batch size determines whether the arithmetic units are saturated or idling between operations.
A filled batch also amortises the fixed cost of streaming weights out of memory across many requests at once.
Across all configurations tested, the penalty at one request per second ran from 17.5 to 36.3 times the saturated cost, collapsing to roughly parity by fifty.
The study’s sharpest observation concerns the tools teams use to plan with: public calculators take utilisation as a figure the user supplies rather than as the variable that governs the outcome, so a utilisation-naive estimate understates true cost by exactly the reciprocal of utilisation.
That systematically oversells self-hosting, most severely for the low-traffic workloads where the case is weakest. How the serving layer converts concurrency into throughput is covered in the post on model serving architecture.
The cheapest operating point is usually out of reach
Those floor figures are measured without a latency bound, which is why they are floors. Put a realistic service level against them and the picture changes.
Take a contract of 300ms P99 time-to-first-token and 50ms P99 time-per-output-token, which is unremarkable for an interactive product.
At twenty-five requests per second the same dense deployment records 259ms and 37.8ms, so it holds. At fifty, where cost reaches its minimum, those tails become 995ms and 71.3ms. The final 14% of cost saving is bought with a near-fourfold increase in tail latency, and it breaches the contract.
An operator bound by that SLA cannot reach the cheap end of the curve at all.
This is why latency targets are the most expensive commitment nobody prices explicitly, and why splitting traffic matters: embedding generation, overnight scoring and evaluation runs carry no such bound and can sit on interruptible capacity at a fraction of on-demand rates.
Most products contain both kinds of traffic and provision them identically.
Contract structure compounds it from the other direction. Reserved capacity carries a substantial discount but bills whether or not the hardware is busy, so reserved capacity paired with spiky traffic funds idle silicon at peak-sized volumes.
For European deployments there is a further dimension, since routing inference outside the EU adds round-trip latency on top of the transfer questions, making the cheaper-looking option slower and more constrained at once.
Those considerations are worked through in the post on data residency in Europe.
The levers that move the number
Once a deployment is running near saturation, model architecture matters less than most procurement conversations assume. Across the three architectures tested, saturated cost converged to within a factor of three, against a load-driven spread of up to 36.
Quantisation moves it more reliably. FP8 raised peak throughput by 31% on the dense model and by 69% and 74% on the two mixture-of-experts models because sparse activation is memory-bandwidth bound and halving parameter precision doubles effective bandwidth.
That produced a result worth pausing on: at saturation, a 30B-parameter sparse model in FP8 came in at $0.209 per million tokens against $0.238 for an 8B dense model.
Active parameter count, rather than total model size, governed the cost. Whether a smaller adapted model reaches a given quality target more cheaply is examined in the post on when PEFT makes sense.
Quality itself is bought in compute, and the usual architectural response is a cascade rather than a larger model everywhere: a cheap first pass over all traffic, then an expensive second pass over the fraction that needs it. That makes the routing threshold a budget decision as much as a quality one.
What to characterise before procuring
Most inference cost estimates go wrong before any hardware is chosen, because the workload was never characterised. Four figures do most of the work:
- Peak concurrent request rate, and the shape of the curve around it – This sets provisioning, and the ratio of peak to mean sets how much idle capacity is being financed.
- The latency percentile you are contractually bound to – P50 and P99 imply different architectures and different bills, and the demo figure is rarely the committed one.
- The input and output token shape of a typical request – Cost per million output tokens behaves differently for short prompts with long generations than for the reverse.
- The quality threshold you can defend, stated as an operating point – A headline accuracy figure cannot be costed, because it does not say where the threshold sits or what the fallback path handles.
The figures above come from list-price on-demand billing with prefix caching disabled and synthetic prompts, which the authors treat as a conservative upper bound on cost.
A production deployment with shared prefixes and reserved pricing would do better. The point is the shape rather than the absolute level, and the shape holds.
Building capacity planning into product work rather than treating it as procurement is set out on the AI product development pillar.
Neurotechnology Cloud designs and operates production serving environments on dedicated EU-resident GPU capacity through its applied AI solutions practice, with throughput and threshold work done before capacity is committed.
If you are sizing an inference workload against fixed latency and quality targets, get in touch with our team here at Neurotechnology Cloud.
