GPU and Compute Economics

What does an accelerator-hour actually buy?#

An accelerator-hour is one GPU or other accelerator available to you for one hour. It is a capacity input, not an output. A working model connects five layers, and value leaks at every join:

contracted capacity → scheduled capacity → productive runtime → accepted output → customer revenue

Maintenance windows, fragmentation, queueing, model loading, failed jobs, retries, low batch density, and rejected outputs each remove a slice of the capacity you paid for. The purchasing model then decides which of those risks you carry:

Google Cloud states that Spot prices are dynamic and deliver 60% to 91% off the corresponding on-demand price for most machine types and GPUs, with smaller discounts on some accelerator families, and that Spot VMs can be preempted at any time (Google Cloud, GPU pricing). AWS charges the Capacity Block reservation fee upfront at purchase, prices it dynamically against supply and demand, and bills the operating system separately (AWS, Capacity Blocks pricing).

Why does this matter to founders?#

It decides whether gross margin scales. AI revenue can grow while margin falls if customers adopt longer contexts, premium models, more retries, or low-density real-time workloads. Per-seat pricing is especially exposed, because usage is unbounded while revenue is not — see Gross Margin.

Capacity is a product promise. A reservation can look expensive at average load and still be necessary for latency, regional residency, or a contractual service level. The right way to record that is an explicit risk value, not a fudge buried in the rate.

Optimisation trades against quality. Batching, quantisation, smaller-model routing, caching, and output caps all raise throughput. They can also reduce accuracy, lengthen tails, or change the failure distribution. Measure accepted outcomes, not benchmark tokens.

Investors will interrogate the bridge. Margin improvement that comes from durable engineering is worth more than margin improvement that comes from a one-off cloud credit or an optimistic allocation. A capacity model makes the difference auditable.

How do you calculate cost per accepted output?#

Start with what you paid for:

available accelerator-hours = accelerator count x hours in period

Then separate the two very different loss mechanisms:

productive accelerator-hours = available hours x scheduled utilisation x runtime success rate

Keep those two factors apart. A scheduling problem is solved by better packing and demand shaping; a runtime-failure problem is solved by engineering. One blended percentage hides which of the two you have.

accepted output = productive hours x throughput per accelerator-hour x acceptance rate

all-in cost = accelerator charges + host CPU/RAM + storage + networking
            + orchestration + observability + directly attributable operations

cost per accepted unit = all-in cost / accepted output

Reservation break-even#

If ancillary costs are identical under both options:

break-even utilisation = reserved capacity cost / (available hours x on-demand rate)

Below that point on-demand is cheaper on a pure cost basis; above it the reservation wins. The decision still needs a value for supply assurance, interruption risk, and engineering complexity — but that value should be written down separately, not smuggled into the rate.

Classify before you buy#

Worked example: does the reservation pay?#

An AI company is deciding how to buy eight NVIDIA H200 accelerators — one p5en.48xlarge — for a month. Rates are US East, verified 13 August 2026: AWS Capacity Block at $54.920 per instance-hour ($6.865 per accelerator-hour); on-demand at $63.296 per instance-hour ($7.912 per accelerator-hour) and spot at $27.209 ($3.401 per accelerator-hour) per the Vantage EC2 price tracker, which is a third-party source rather than an AWS rate card.

available accelerator-hours = 8 x 730 = 5,840
reserved accelerator cost   = 5,840 x $6.865 = $40,091.60
host, storage, network, observability, on-call = $14,600
total monthly cost                             = $54,691.60

The workload produces 1.8 million accepted output tokens per productive accelerator-hour.

Nothing about the hardware price changed across those rows. Utilisation alone moves unit cost by 2.4x.

Now test the reservation against on-demand.

break-even utilisation = $40,091.60 / (5,840 x $7.912) = 86.8%

This is the finding that most capacity models get wrong. Because the current AWS Capacity Block rate is only 13.2% below on-demand for this instance, the reservation has to run at almost 87% productive utilisation before it wins on price alone. At a realistic 65% it is over $10,000 a month more expensive.

That does not make the reservation wrong — it makes the justification different. If on-demand p5en capacity cannot actually be obtained during peak demand, the reservation is buying supply assurance, and the right way to write it up is: "we are paying $10,058 a month for guaranteed access." That is a defensible sentence. "The reservation is cheaper" is not.

Acceptance rate compounds everything. Hold utilisation at 65% and vary the share of output the product accepts:

100% accepted →  $8.00 per million accepted tokens
 90% accepted →  $8.89
 75% accepted → $10.67

A 25-point drop in acceptance costs more than the entire reservation discount.

Caveat on this model. The on-demand and spot rates come from a third-party tracker, and spot pricing moves continuously; the Capacity Block rate is AWS's own and AWS notes prices are updated regularly, with the next scheduled change in October 2026. Re-pull all three before signing anything. The model also assumes ancillary cost is identical across purchasing options, which is true for storage and network and false for the engineering time that checkpointing for spot requires.

Related concepts#

• Inference vs. Training Costs — separate recurring delivery cost from model-development investment.
• Token Economics and Metering — attribute compute consumption to customers and tasks.
• AI-as-a-Service — connect capacity to packaging and service levels.
• AI Unit Economics and Gross Margins — carry cost per accepted unit into cost of revenue.
• Hardware-as-a-Service — compare capacity ownership with a service contract.
• Usage-Based Pricing — charge for a controllable unit of consumption.
• Pricing Metric / Value Metric — avoid exposing a technical input as the customer meter.
• Economies of Scale — test whether serving cost actually falls at higher volume.

Sources#

1. Amazon Web Services, Amazon EC2 Capacity Blocks for ML pricing, accessed 13 August 2026. Per-instance and per-accelerator reservation rates for P5, P5e, P5en, P6-B200, P6-B300 and Trainium instances; the upfront reservation fee; the separate operating-system charge; and the note that prices are updated regularly with the next change scheduled for October 2026.
2. Vantage, p5en.48xlarge pricing and specs, price snapshot 13 August 2026. Third-party tracker used for the on-demand and spot comparison rates; not an AWS rate card, and should be re-checked against the console before any commitment.
3. Google Cloud, GPU pricing, accessed 13 August 2026. Sustained-use and committed-use discount mechanics, the requirement to attach an unmodifiable reservation for GPU committed-use discounts, and the 60–91% Spot discount range with preemption at any time.
4. NVIDIA, NVIDIA Announces Financial Results for Fourth Quarter and Fiscal 2026, February 2026. Fiscal 2026 revenue of $215.9 billion, Data Center revenue growth, and GAAP gross margin of 71.1% alongside the Hopper-to-Blackwell transition.

Price-freshness note: Every compute rate on this page was checked on 13 August 2026 against the source named beside it. AWS states Capacity Block reservation prices are updated regularly and are next scheduled to change in October 2026, and spot prices move continuously — re-verify before using any figure in a board pack, a customer quote, or a capacity commitment.