LLM token pricing explained: how model costs actually work

Every AI provider publishes a pricing page listing a dollar amount per million tokens, and every one of those pages tells you less than it looks like it does. Understanding what actually drives your bill requires going one level deeper than the rate card.

What a token actually is

A token is roughly a chunk of text, sometimes a whole word, sometimes part of one, that a model processes as a unit. Pricing is quoted per million tokens because the numbers involved are large enough that per token pricing would be an awkward unit to reason about. The important detail most people miss at first: input tokens, the text you send the model, and output tokens, the text it generates back, are priced separately and almost always at different rates, with output tokens typically costing several times more than input tokens for the same model.

Why the sticker price is only part of the real cost

• Prompt cachingWhen consecutive requests share a stable prefix, such as a system prompt or a retrieved document that does not change between calls, providers let you skip reprocessing it. Cached input tokens can cost a small fraction of the standard rate, and providers vary meaningfully in how aggressive that discount is.
• Batch processingNearly every major provider offers a batch endpoint priced well below the live, real time rate in exchange for an asynchronous response window. Anything that does not need a sub second answer, evaluation runs, enrichment jobs, classification at scale, is a candidate for this discount and many teams never turn it on.
• Reasoning tokensModels that think step by step before answering generate internal reasoning tokens that are billed as output, even though the user never sees them. This makes the effective cost per useful, visible answer meaningfully higher than the headline output rate suggests for reasoning heavy models.

Why comparing sticker prices across providers is misleading

A model that looks twice as expensive per million tokens on paper can end up cheaper in practice once caching, batching, and actual task performance are accounted for, because a weaker but cheaper model might need multiple retries or a longer prompt to do the same job as a stronger one in a single pass. Comparing AI model costs on rate card price alone, without accounting for how many tokens a given task actually requires on each model, routinely produces the wrong answer.

What actually determines your bill

Your real monthly cost is a function of four things multiplied together: the rate per token, the number of tokens per request, the number of requests, and which discount mechanisms you have actually enabled. Most organizations can name the first variable easily and have almost no visibility into the other three, which is why the invoice so often surprises them.

Where to look instead of a static price table

Published rate cards go stale the moment a provider adjusts pricing, and most providers now adjust pricing more often than once a quarter. A living view of current pricing across every provider and host you actually use, updated as prices change rather than on a fixed publishing schedule, is the only reliable way to know what a switch would actually save you today, not what it would have saved you last quarter.