The 20× Inference Bill Nobody Is Watching

Somewhere in your cloud spend there is a line item that grew 400% last year and nobody owns. It is not compute or storage. It is inference — and unlike every other infrastructure cost, it scales with usage in a way that punishes exactly the products that are working.

The uncomfortable arithmetic looks roughly like this. At a million conversations a month, a hosted frontier model runs somewhere in the region of $15,000 to $75,000 per month. A fine-tuned small model serving the same volume from your own infrastructure lands closer to $150 to $800. That is a 20× to 100× difference — and for a large share of what production systems actually do, the small model performs the task as well or better.

This is not an argument against frontier models. It is an argument against using one model for everything, which is what most systems do because it is what the first prototype did.

"Small" stopped meaning "weak"

The reason this became a live question in 2026 is that the capability floor moved. Small used to mean under a billion parameters and noticeably worse at everything. The productive band is now roughly 3B to 14B — models that run on modest hardware without a GPU cluster, and that handle classification, extraction, routing, summarisation, formatting and structured output at a quality that would have required a frontier model in 2024.

Both platform vendors have been pushing this from opposite directions — efficient model families aimed at on-device inference on the desktop side, lightweight variants and native inference paths on mobile. The effect is the same: useful intelligence has moved closer to the user, and the default assumption that every token must cross the network to a datacentre no longer holds.

The pattern: route, don't choose

The design that has emerged is tiered. A small model handles the routine steps locally. A frontier model is escalated to only when the task genuinely needs it. Reported splits cluster around 80–90% of steps handled locally, which means the cloud bill is the remaining 10–20% — a fifth to a tenth of the cloud-only column.

The thing to notice is that most production AI workloads are not one task. A support agent might:

• Classify an incoming message by intent — trivial, small model
• Extract the order ID and dates — trivial, small model
• Decide which knowledge base section is relevant — trivial, small model
• Reason about an unusual multi-part complaint with conflicting policy — frontier model
• Draft the reply in the right register — depends; often small, fine-tuned
• Format the structured output for the ticketing system — trivial, small model

Five of six steps are mechanical. Sending all six to a frontier model is paying research-grade reasoning rates to parse a date.

Three benefits that aren't the bill

Latency you control

A local model has no network round trip, no queue and no noisy-neighbour variance. For anything interactive, the p99 matters more than the median, and hosted p99s are not yours to fix. Steps that run in tens of milliseconds locally change what interactions are feasible.

Data that never leaves

On-device or on-premises inference keeps sensitive input inside your boundary. For healthcare, finance, legal and anything under strict residency rules, this often converts a project from "needs a year of review" to "runs on infrastructure we already have accredited". That is frequently a bigger unlock than the cost saving.

A dependency you own

Self-hosted weights do not get deprecated on someone else's schedule, do not change behaviour overnight, and do not rate-limit you during a traffic spike. If you have ever had a model version retired underneath a working system, you know what this is worth.

Where small models are the wrong answer

The case is real, and it is not universal. Reach for the frontier model when:

• The task needs multi-step reasoning over a novel problem. This is the widest remaining gap and it is not close.
• You need broad world knowledge that cannot be supplied by retrieval.
• The task is open-ended and you cannot enumerate what good looks like — which also means you cannot fine-tune for it.
• Volume is low. At ten thousand calls a month, the difference is a rounding error and the engineering time to build a routing layer is not. Optimise this when the bill justifies it.
• You have no evaluation harness. Without one you cannot tell whether the small model is doing the job, and you will find out from a customer.

And count the real total: self-hosting means GPUs, serving infrastructure, monitoring, a fine-tuning pipeline and someone on call. That overhead is fixed, which is exactly why the economics flip at volume and stay bad below it.

How to find out what you'd save

This is a measurable question, not a philosophical one. In about a week:

• Instrument by step, not by month. Break your spend down by which call in your pipeline made it. Most teams doing this for the first time find one or two steps carrying the majority of the bill, and they are rarely the interesting ones.
• Take the top step and sample real traffic. A few hundred production inputs with the outputs you currently ship.
• Run a small model against the same inputs and compare. For classification and extraction you can score this automatically.
• If quality holds, shadow it in production — small model runs alongside, output logged but not served — until you trust the numbers.
• Then switch that one step, keeping the frontier model as the escalation path for low-confidence cases.

One step at a time, with an evaluation set at every stage. The teams that get burned are the ones that swap the whole pipeline at once and discover the regression in aggregate.

The bigger shift

For two years the default architecture was: send everything to the biggest available model and let capability absorb the design problem. That worked while volumes were small and budgets were experimental. Both of those conditions have ended — boards are asking for demonstrated return and CIOs are expecting cuts against missed targets.

What replaces it is unremarkable engineering: profile the workload, match each step to the cheapest thing that does it correctly, measure continuously, escalate when needed. We have been doing this with compute for decades. Inference is just the newest resource to run the exercise on.

We profile AI workloads, build the routing layer and the evaluation harness that makes switching safe. Send us your inference bill.