This is the first essay in the Economics & Deployment pillar. It assumes you have read What is a token, really?; if you have not, do that first — the unit matters here.

Two bills, not one

The cost of an AI feature splits cleanly into two pieces:

  • Training cost. Paid once per model, by the lab that built it. A frontier-scale training run today involves tens of thousands of accelerators running for weeks or months. Public figures and third-party trackers like Epoch AI estimate the largest training runs at hundreds of millions of dollars in compute alone.1 The Chinchilla paper is the canonical reference for how labs decide how much data and compute to spend on a given model size.2 The earlier Kaplan scaling-laws paper set up the framework Chinchilla corrected.3
  • Inference cost. Paid every time a user sends a prompt. This is the cost that shows up on your API bill.

If you are building on top of someone else’s model, only the second cost is yours. But the first cost shapes the second — it determines how big the model is, how it was tuned, and what the provider needs to charge to recoup the spend.

What you are paying for, per token

When an API charges you, say, $3 per million input tokens, that price is covering:

  1. A slice of GPU time. The model has to do roughly two floating-point operations per parameter per token. A 70-billion-parameter model doing inference on a thousand tokens is on the order of 140 trillion floating-point operations. Modern accelerators do that in well under a second, but the cost of that second of accelerator time is what dominates.
  2. Electricity. Datacenters running these workloads draw real grid power. The IEA’s 2024 electricity outlook flagged AI workloads as a meaningful new line on global demand projections.4 Earlier work by Patterson and colleagues quantified emissions from individual training runs.5
  3. Amortized training, and margin. Some fraction of every inference dollar pays back the training run, the research staff, and the provider’s profit.

Why the price keeps falling

Inference costs per quality-adjusted token have fallen roughly an order of magnitude per year since 2023, driven by three forces stacking on top of each other:

  • Smaller models that match older big ones. Distillation, better data, and the lessons of Chinchilla mean a 2026 8B-parameter model can do what a 2023 70B model could.
  • Better serving stacks. Speculative decoding, paged attention, and better batching squeeze more tokens per GPU-second out of the same hardware.
  • More competition. When a capability becomes available from three providers instead of one, the price floor drops to whichever is willing to lose money for share.

What this changes

The applications that barely did not pencil out a year ago — agents that take many model calls per user action, products that summarize every email, voice interfaces that run continuously — are the ones whose unit economics quietly flip first.

If you are deciding whether to build something, the right question is not “can a model do this today?” but “can a model do this at the price it will cost in eighteen months?” That is usually about ten times less than today.

Footnotes

  1. Epoch AI, Trends in Machine Learning. ↩

  2. Hoffmann et al., Training Compute-Optimal Large Language Models, 2022. ↩

  3. Kaplan et al., Scaling Laws for Neural Language Models, 2020. ↩

  4. IEA, Electricity 2024. ↩

  5. Patterson et al., Carbon Emissions and Large Neural Network Training, 2021. ↩

Citations

  1. Training Compute-Optimal Large Language Models (Chinchilla) — Hoffmann et al., 2022 · accessed May 17, 2026
  2. Scaling Laws for Neural Language Models (Kaplan et al., 2020) · accessed May 17, 2026
  3. Carbon Emissions and Large Neural Network Training (Patterson et al., 2021) · accessed May 17, 2026
  4. Electricity 2024 — IEA (Wayback Machine snapshot, 2026-05-07) · accessed May 17, 2026
  5. Epoch AI — Trends in Machine Learning · accessed May 17, 2026

More in Economics & Deployment