Why LLM inference costs grow faster than usage, and how to cut them

By the Radium team · Last updated November 2026

LLM inference costs grow faster than usage because you pay for tokens, not users, and the tokens behind each request multiply as features are added. In our worked example, the same 300,000 conversations a month cost $990 as a prototype and $33,302 once the product is fully built, with no new users at all.

Why LLM inference gets more expensive as usage scales

Usage is not tokens

The bill is metered in input and output tokens, so user counts and request counts understate real growth. Inference dominates: AWS estimates, as cited by TensorMesh, that it can account for up to 90% of infrastructure spend for ML applications, and the FinOps Foundation reports 80 to 90% of GenAI spend landing on inference. CloudZero reports that heavy agentic users typically burn $100 to $250 a month on inference alone. As a small anchor, nOps notes that a 4,000-token prompt on a top-tier model runs about $0.12 per request, so 100,000 requests a month is about $12,000 of input cost before any output. Our earlier article on the hidden costs of AI covers the broad picture. This one goes deeper on the arithmetic.

Where the extra tokens come from

In the worked example below, features land one step at a time on flat traffic. A longer system prompt with few-shot examples takes the prototype to the production prompt stage: input from 400 to 2,900 tokens per conversation, bill from $990 to $2,490. Retrieved chunks for RAG push input to 5,900 tokens and the bill to $4,290. Conversation history is the cliff edge. Resending full history across 5 turns carries 31,200 input tokens per conversation and jumps cost from 4.3 to 22.7 times the prototype bill. Agent tool loops pile on: 2 tool calls turn into 7 model calls and 44,480 input tokens per conversation, and TrueFoundry notes that agentic workflows multiply inference spend because one task triggers several model calls. Reliability work closes it out: a guardrail check plus an 8% full-retry rate reach 8.64 call equivalents per conversation. Gartner forecasts, as cited by TensorMesh, that inference cost per agentic workflow will rise more than fivefold through 2028.

Output tokens cost more and compound

Output is billed at 3 to 5 times input, and the gap widens at the top of the range. TensorMesh observes 5 to 6 times input for flagship output at short context and about 4 times in long-context tiers, over short-context input spanning $0.20 to $4.00 per 1M tokens. Both sides grow together in multi-turn work, because every turn generates fresh output while resending input. Falling list prices do not rescue the bill. Telnyx notes that the Stanford AI Index found the cost of a GPT-3.5-level query fell more than 280-fold between November 2022 and October 2024, yet the per-workflow forecast above still points up. Tokens per workflow grow faster than prices fall.

A worked example: one workload, six stages

The setup

One support and copilot workload, flat at 300,000 conversations a month in every stage, priced on Claude Sonnet 5.5 at Anthropic's list rates: $2 input and $10 output per 1M tokens, with prompt-cache rates from the same table. Stages add the features in the order products ship them: prototype prompt, production prompt, retrieval, multi-turn, tool use, reliability. Failure handling arrives last: a guardrail check on Claude Haiku 5.5 at $0.10 input and $0.50 output per 1M tokens, plus 8% full retries.

Reading the table

Traffic is flat at 300,000 conversations a month in every row, and cost alone moves from $990 to $33,302, or 1.0x to 33.6x. Input tokens per conversation grow from 400 to 49,118.4 while output grows from 250 to 1,533.6: most of the waste is context, not answers. One decision doubles the bill near the start, since a fuller production prompt alone is 2.5x. History is the cliff edge at 4.3x to 22.7x, and retries plus the guardrail check add the last visible chunk, 31.1x to 33.6x. The climb is optional: the levers below take the same workload from $33,302 to $4,878.

‍

‍

Assumptions: 300,000 conversations a month, constant; S1 to S5 on Claude Sonnet 5.5 at $2 input and $10 output per 1M; S6 adds a guardrail call on Claude Haiku 5.5 and 8% full retries, modeled as all usage times 1.08. Swap in your own token counts, call counts and traffic, and the same arithmetic gives your bill.

How enterprises reduce GenAI costs in production

Instrument first, then cut

Measure before optimizing. Microsoft's guidance says the first 80% of cost wins come from tagging everything from day one, giving one owner a weekly cost review, and applying the levers in order. For a ceiling on the combined move, NeuralTrust finds that model routing, prompt caching and output-length limits together cut inference bills 40 to 70%.

The cost levers table

The levers below are ordered by combined impact against effort. Modeled savings come from applying each one cumulatively to the $33,302 workload above.

‍

‍

What the cumulative math shows

Applied cumulatively to the S6 bill, four modeled levers move $33,302 to $4,878. Prompt caching is the largest single lever, worth $19,424 in one step. History trimming takes $1,788 more, model routing $6,836, and output caps $376.

‍

‍

Test one sensitivity before you route: if conversations that move to the smaller model lose cache treatment and are billed at plain input and output rates, cumulative saving drops from 84.2% to 82.6%, so keep caching in place when you route. Two cross-checks from practice: CloudZero reports that ProjectDiscovery cut its LLM spend 59 to 70% by lifting cache hit rates from 7% to 84%, and NeuralTrust reports 75 to 85% cost reduction for batch-heavy async work. Routing design is a topic in itself: see The Business Case for Matching AI Models to Workloads for the method.

Cutting inference costs without going direct to OpenAI or Anthropic

Compatible API providers

One option is to point the same OpenAI-style or Anthropic-style calls at a different host. Supplier competition is real: Inworld lists public NVIDIA B200 rates spanning $3.49 to $14.24 per GPU-hour across clouds in April 2026, and that spread reaches token prices. Providers differentiate on discounts: Telnyx lists cached input at up to 88% off, and Anthropic and OpenAI both take 50% off batch work. Options include Together AI, Fireworks AI, Radium and others, with no ranking implied. Switching on a compatible API is small work: change the base URL, key and model name, which the switching guide walks through. Trade-offs to weigh: model availability, latency, SLA differences and vendor review, and check SOC 2 status before moving production traffic.

Routing and gateways

A routing layer sends each request to the cheapest model that can do it, and you stay on your current provider. Microsoft recommends routing at least 60% of traffic to cheaper models without quality regression. TrueFoundry highlights semantic caching and model routing as the primary controls on token spend, and NeuralTrust measures 40 to 70% off the bill when routing, caching and output caps land together. Trade-offs: evals take build time, requests need difficulty grading, and the gateway is one more service to run. In the modeled case, routing is the step that takes cumulative savings to 84.2%.

Open-weight models, served or self-hosted

The third option is serving open-weight models, either through a provider that hosts them or on GPUs you rent yourself. Self-hosting means fixed cost: nOps puts one H100 at $2 to $3 per hour, about $1,500 to $2,200 a month, and a 4 to 8 GPU production cluster at roughly $6,000 to $17,000+ a month. The catch is utilization. The FinOps Foundation sees many workloads at 15 to 30% of capacity, with underutilization reaching 70 to 85%, and GMI Cloud puts GPUs without proper batching at 20 to 40%. nOps notes batching lifts utilization to 70 to 90%, but batching and capacity planning then sit with your team. GMI Cloud reports quantization cutting memory cost 50 to 75%, continuous batching delivering 10 to 20 times, up to 23 times, throughput over naive processing, and cost-effective deployments matching model size to GPU class. Fair summary: self-hosting wins at steady high utilization with real MLOps in place, and spiky or uncertain traffic usually favors pay-per-token APIs. A Systems View of Enterprise AI covers the infrastructure decisions that choice forces. The three options stack: teams can keep OpenAI or Anthropic for some work and still cut total spend.

Where Radium fits

Radium serves open-weight models (Hal 1.0, Clarke 1.0 and Tycho 1.0) tuned and run on infrastructure Radium owns and operates, priced per token on the pricing page. Teams moving Production AI workloads to Radium can cut inference spend by up to 50%, and prompt caching saves up to 90% on repeated prompts, which is the largest lever in this article. Switching is a change of base URL, key and model name on the OpenAI-compatible or Anthropic-compatible endpoints; the switching guide covers the steps. New accounts get $25 in credit, no card required, and the docs cover setup. Radium is SOC 2 compliant.

Sources

• CloudZero: Inference cost
• Inworld: LLM inference cost at scale
• Telnyx: Inference cost
• TensorMesh: LLM inference cost
• NeuralTrust: LLM cost reduction guide
• nOps: LLM cost optimization tips
• FinOps Foundation: Optimizing GenAI usage
• Maxim: Reducing your OpenAI and Anthropic bill with semantic caching
• TrueFoundry: AI cost optimization strategies
• Microsoft Learn: AI cost optimization
• GMI Cloud: How to scale LLM inference without breaking your budget
• Anthropic: Model pricing
• Anthropic: Prompt caching