Almost every cost estimate for a conversational agent is built the same way and is wrong by an order of magnitude in the same direction. Someone takes the length of a typical question, adds the length of a typical answer, multiplies by the expected number of conversations, and multiplies that by a published per-token rate.
The error is not in the arithmetic. It is that the question and the answer are, in a production agent doing anything useful, roughly one percent of the tokens you are billed for.
What follows is a model of what you actually pay for when you operate agents for twenty clients, denominated in tokens rather than currency, with the multipliers that hide inside it and the one term whose growth is not linear.
Why the model is in tokens
Because per-token prices move faster than the model does, and in both directions.
Stanford HAI's 2025 AI Index recorded that the cost of querying a model scoring the GPT-3.5 equivalent of 64.8 on MMLU "dropped from $20.00 per million tokens in November 2022 to just $0.07 per million tokens by October 2024" — a more than 280-fold reduction in roughly eighteen months. Any cost model written in dollars in 2023 was off by two and a half orders of magnitude by 2024, and any conclusion drawn from it about what to build was worthless within a year.
The reflex is to conclude that inference is trending to free and stop modelling. The 2026 AI Index, published 13 April 2026, is a corrective: global corporate AI investment reached $581.7 billion in 2025, up 130% on the prior year, with AI data centre power capacity at 29.6 GW. Capital at that intensity is not a signal that the binding constraint has gone away — it is a signal that capacity is being bought, and capacity that is bought gets priced.
So the durable form of the model is a token count. Multiply by whatever your provider charges you this quarter. The structure below does not change when the rate does.
The arithmetic of one turn
A conversational turn is not one message. It is a prompt assembled from six parts, five of which the user never sees:
| Component | Typical size | Sent how often |
|---|---|---|
| System instructions | 400–800 tokens | Every turn, unchanged |
| Rolling conversation summary | ~300 tokens | Every turn, once summarisation is active |
| Recent history window | up to ~1,500 tokens | Every turn |
| Retrieved chunks | 5 × ~500 = ~2,500 tokens | Every turn that touches the knowledge base |
| The user's message | ~30 tokens | Every turn |
| Completion | ~200 tokens | Every turn |
These sizes are ours, from deployments we operate, not a published benchmark — treat them as a shape rather than a constant. What moves them most is chunk size and how many chunks the retrieval step returns, which is a configuration decision, not a property of the model.
A mid-conversation turn is therefore around 4,700 input tokens and 200 output tokens. The user's question is 0.6% of the input. The retrieved context is a little over half of it.
At six turns per conversation, a conversation costs roughly 28,000 input and 1,200 output tokens. Twenty clients at 1,500 conversations a month each is 30,000 conversations, or on the order of 850 million input tokens per month across the fleet.
That is the number people build a business case on. It is also the floor, because four multipliers sit on top of it.
Multiplier 1: the tool loop, ×2 to ×4
A turn that calls a tool is not one model call. It is a call that decides which tool to invoke, then a call that interprets the result — and if the agent chains two lookups, or retries one that returned nothing useful, it is four or five. Each of those calls re-sends the entire prompt, now grown by the accumulated tool results.
On a tool-heavy agent this is the single largest correction to the naive estimate, and it is invisible in per-conversation logs that only count user turns. Measure model calls, not messages. The ratio of the two is the multiplier, and on agents that do real work we see it between two and four.
Multiplier 2: the checks are model calls too
Semantic guardrails — scope enforcement, injection classification, custom business rules — are themselves inference. Their prompts are short, which makes each one cheap and makes the total easy to dismiss. Seven checks per turn on short prompts is small next to a 4,700-token reasoning call, but it is a fixed addition to every turn including the ones that were going to be cheap, and it lands on the fast path where you also pay in latency.
The optimisation is ordering, not removal: deterministic checks before semantic ones, so a regex rejects what would otherwise have cost a model call.
Multiplier 3: summarisation is a spike, not a saving
Compression is not free — the summariser reads the window it is compressing. Every compaction is an extra call over one to two thousand tokens.
It is still the highest-return line in the model, for a reason that is worth stating precisely rather than as a rule of thumb. Without a capped history window, a conversation's total token cost is quadratic in its length. Turn n re-sends turns 1 through n−1, so the total across T turns goes as the sum of 1 through T — order T². A twenty-turn conversation does not cost three times a six-turn one; it costs about eleven times as much.
A rolling summary caps the per-turn prompt at a constant, which converts the total to order T. That is not a percentage saving. It is a change in the growth curve, and it is why a support agent handling long threads is either summarising or is a budget incident waiting for a chatty customer.
Multiplier 4: the parts that bill while nothing happens
Four lines that do not correlate with conversation volume at all:
- Ingestion embeddings. Every document is embedded once at upload. Cheap per document, real for a client arriving with a 4,000-page corpus.
- Re-embedding. Change your chunking strategy, or switch embedding models, and every document in every knowledge base is re-embedded. This is a migration priced as a config change, and it is the reason the embedding model is the stickiest choice in the stack.
- Scheduled source syncs. A knowledge base connected to a document store re-embeds what changed, on a schedule, whether or not anyone talked to the agent.
- Failover. When a provider degrades and traffic shifts to the next entry in a chain, the unit cost of every conversation changes to whatever the fallback costs — silently, for as long as the incident lasts. That is a resilience mechanism with a billing side-effect, and it belongs in the cost model because it fires precisely when nobody is looking at the cost model.
An idle client is not a free client. It is a smaller number that is not zero, which matters when you are pricing a tier that expects low usage.
Sensitivity: which knob actually moves the number
Four inputs, ranked by how much the fleet total moves when you change them.
| Knob | Effect on total | Realistic range |
|---|---|---|
| Retrieved chunks per turn | Linear and dominant — retrieval is over half the input | 5 → 3 chunks removes roughly a quarter of all input tokens |
| History cap | Changes the growth curve, T² → T | The difference between bounded and unbounded cost per conversation |
| Model per reasoning level | Linear, large, and free to change | Classification and summarisation at the cheap level; only the answer at the expensive one |
| Tool hops per turn | Multiplicative | ×2 to ×4, and the hardest to reduce without changing behaviour |
The ordering is the finding. Teams reach for the third row first because switching models is a visible decision with a number attached to it, and it is genuinely worth doing. But halving retrieved chunks is usually a larger saving than changing model, costs nothing but an evaluation run, and frequently improves answer quality — five mediocre chunks dilute the two good ones.
The second row is the one that turns a cost problem into a cost incident, and it is the cheapest to fix.
What this means for how you charge
The commercial conclusion follows from the arithmetic rather than from any pricing theory.
Do not price per seat or per agent. Neither correlates with cost. A client with three agents and long retrieval-heavy threads costs multiples of a client with fifteen agents answering FAQs, and a per-agent price bills the second client to subsidise the first.
Do not cap on messages alone. Message count is a proxy for cost only if tokens per message are similar across clients, and they are not — retrieval configuration varies per client by design. A message cap set from your average client is generous to your heaviest one and restrictive to your lightest.
Cap on spend, with a soft threshold below it. A hard ceiling per billing cycle bounds the exposure; a soft threshold at some percentage of it is what gives you a conversation with the client before the ceiling stops their agents mid-month. Two numbers, and the second one is what keeps the first from being a support incident.
And the reason any of this is tractable at twenty clients rather than twenty subscriptions is that the aggregate exists at all. Per-account platforms produce twenty invoices and no fleet view; the question "what did inference cost us last month" becomes archaeology across twenty billing portals. One installation makes it a query, which is a precondition for pricing anything accurately.
Five things to measure before you quote a client
Numbers, not opinions. Every one is available from run-level telemetry that you should already be keeping.
- Mean input tokens per model call, split by node type. If reasoning calls are under 2,000, you are probably under-retrieving; if they are over 8,000, find out which component is the outlier.
- Model calls per user turn. This is your tool-loop multiplier. Anything above 4 needs an explanation.
- Turns per conversation, at the 90th percentile, not the mean. The mean hides the long threads, and the long threads are where the T² term lives.
- Share of turns that hit retrieval. If it is 100%, some of those turns did not need context, and each one is carrying 2,500 tokens it did not use.
- Tokens per conversation by client. Then look at the spread, not the average. The ratio between your heaviest and lightest client is the single number that tells you whether one price can cover both.
Run those five against a month of real traffic before agreeing to a fixed monthly fee for anything. The arithmetic above will get you within a factor of two of the answer. Your own telemetry gets you the rest of the way, and the gap between the two is where the margin is.
