Inference vs. Training Costs

What is the difference between training and inference cost?#

Training and inference are stages of the same production system, with very different cash profiles.

Three consequences follow. First, most startups buy pre-trained capability and own only the inference curve. Second, model-development choices change that inference curve: the Chinchilla result showed that under a fixed training-compute budget, a 70-billion-parameter model trained on substantially more data outperformed a much larger model — and a smaller model is cheaper to serve for its entire life (Hoffmann et al., 2022). Third, the boundary is no longer clean. Reasoning systems spend compute again at request time, so "the model" is really a policy for allocating compute across training, routing, and each live task.

Why does this distinction matter to founders?#

Pricing and gross margin. If inference is a meaningful variable cost, a flat subscription hides loss-making power users. If most of your cost is reserved capacity, pure per-token pass-through underprices the availability and latency guarantee you are actually selling. The price architecture should track the dominant cost driver without exposing customers to a meter they cannot predict — see Pricing Metric / Value Metric.

Build, buy, and model choice. Fine-tuning can shorten prompts, raise task success, or let a smaller model do the job. It is economically justified only when those recurring gains exceed the full cost of data, experiments, evaluation, deployment, and ongoing maintenance. The comparison is risk-adjusted lifecycle cost for equivalent outcomes, not one invoice against another.

Fundraising. Investors want to know whether AI cost falls with routing, caching, and volume, or grows with every user action. A credible deck separates five things: model-development investment; variable inference cost per successful task; fixed serving and reliability cost; support and human-review cost caused by model failures; and margin sensitivity to longer outputs, retries, and vendor price changes.

How do you calculate cost per successful task?#

For a period with N successful tasks:

total AI cost = amortized training + fixed serving + variable inference + expected failure cost

For API-priced models, the variable model layer is no longer a single rate:

variable model cost = (uncached input x input rate)
                    + (cache reads x cache-read rate)
                    + (cache writes x cache-write rate)
                    + (output tokens x output rate)

Then normalise to outcomes, not calls:

inference cost per successful task
  = (model + retrieval + tools + guardrails + retries + observability) / successful tasks

The denominator matters more than founders expect. A workflow that costs $0.01 per call, averages six calls, and succeeds 80% of the time costs at least $0.01 x 6 / 0.80 = $0.075 per successful task before tools and human review — 7.5x the naive per-call figure.

For owned or reserved capacity, add the idle you paid for:

serving cost per task = (reserved compute + orchestration + operations) / successful tasks
                      + burst cost per task

When does a fine-tune pay for itself?#

Let T be the incremental cost of fine-tuning and productionising a model, and s the recurring net saving per successful task against the best alternative:

break-even tasks = T / s

Use a net saving that already subtracts evaluation, retraining, monitoring, and any quality loss. Discount future savings if break-even takes years, and cap the horizon at the point where the base model will be superseded anyway. A break-even that lands beyond your model-refresh cycle is not a break-even.

A model-selection scorecard#

Choose on the efficient frontier: an option is dominated only if another is cheaper, faster, and at least as reliable.

input  = 900 / 1,000,000 x $1.00 = $0.00090
output = 300 / 1,000,000 x $5.00 = $0.00150
total                            = $0.00240

input = (900 x 0.40 x $1.00 + 900 x 0.60 x $0.10) / 1,000,000 = $0.000414
per-task model cost = $0.000414 + $0.00150 = $0.001914   (down 20.3%)
monthly total = ($0.001914 x 2,000,000 x 1.25) + $6,000 + $10,000 = $20,785

break-even = $120,000 / $0.004 = 30,000,000 successful tasks
at 2,000,000 tasks/month = 15 months
at 5,000,000 tasks/month = 6 months

Related concepts#

• Token Economics and Metering — build the auditable usage ledger that produces the cost figures above.
• GPU and Compute Economics — model reserved capacity, utilisation, and cost per accepted output.
• AI Unit Economics and Gross Margins — connect cost per task to cost of revenue and contribution.
• AI-as-a-Service — turn the compute stack into a commercial offer with service levels.
• Copilots vs. Agents — understand why autonomy multiplies inference and control cost.
• Usage-Based Pricing — align price with a measurable value or cost driver.
• Credits and Drawdown Models — translate heterogeneous AI usage into a customer-facing balance.
• API-as-a-Product — design the contract, reliability, and developer experience around programmatic usage.
• Economies of Scale — test whether serving cost actually falls at higher volume.
• Data Moats — decide whether the capability a fine-tune buys is defensible.

Sources#

1. Jordan Hoffmann et al., "Training Compute-Optimal Large Language Models", arXiv:2203.15556, March 2022. The Chinchilla result on allocating training compute between model size and data, and why a smaller compute-optimal model is cheaper to serve.
2. Anthropic, Claude Platform pricing documentation, accessed 13 August 2026. Per-model input, cache-write, cache-read, and output rates; the 50% Batch API discount; cache multipliers of 1.25x, 2x, and 0.1x; and the note that Claude 4.7 and later use a tokenizer producing approximately 30% more tokens for the same text.
3. OpenAI, API pricing, accessed 13 August 2026. Standard, Batch, Flex, and Fast-mode rates per million tokens, including short- and long-context bands and cache-write pricing on the GPT-5.6 family.
4. Google, Gemini Developer API pricing, accessed 13 August 2026. Paid-tier input and output rates by model and service tier, and the higher rate band for prompts above 200k tokens on Gemini 3.1 Pro.
5. Amazon Web Services, How tokens are counted in Amazon Bedrock, accessed 13 August 2026. Model-specific output-token burndown rates and the distinction between quota consumption and billable usage.

Price-freshness note: Every rate quoted on this page was checked against the provider's own documentation on 13 August 2026. AI rate cards change on the provider's schedule, not yours — re-verify before using any figure in a board pack, a customer quote, or a pricing decision.