Most software features cost roughly the same to run whether a customer uses them once a month or fifty times a day. LLM features don't work that way. Every request has a cost, longer requests cost more, and a small group of heavy users can account for a large share of the bill.
I learned to take this seriously at WiZR, where we were building content generation for B2B clients. A feature that looked affordable in testing could look very different once one enterprise team started using it all day. So before pricing anything, I wanted a rough model of what each user would actually cost us.
Start with one request
Work out the cost of a single typical request. You need:
- Input tokens: the instructions, any retrieved context, conversation history, and the user's input.
- Output tokens: the length of the response.
- The provider's price per token for input and output, which is usually different. 1
- Any extra calls: retrieval, a second model checking the first, retries.
The part people often forget is the context. A short user question can carry a long system prompt and several retrieved documents with it. Input tokens frequently outweigh output tokens by a lot.
Then work out a user
Cost per request times requests per user per month. The catch is that "requests per user" is a distribution, and a lopsided one.
| Usage group | Requests per month | Share of users | Share of cost |
|---|---|---|---|
| Light | 10 | 60% | 12% |
| Regular | 50 | 30% | 30% |
| Heavy | 300 | 10% | 58% |
These numbers are made up, but the shape is common. A tenth of users can drive more than half of the cost. If your pricing assumes everyone is "average," the heavy group is where you lose money.
Find the expensive patterns
Once real usage data comes in, look at what's driving cost:
- Long conversation history being resent with every turn.
- Large retrieved context where most of the text isn't used.
- Regeneration loops, where users keep retrying because the first answer wasn't good enough.
- Automated or bulk use, where one integration fires thousands of requests.
The regeneration one is interesting because it's a quality problem showing up as a cost problem. Better first answers can cut cost and improve the experience at the same time.
The levers, roughly in order
- Trim the prompt and context. Often the cheapest change with no quality loss.
- Cache repeated work. If many users ask similar things against the same material, some work can be reused.
- Use a smaller model for simpler steps. Classification or routing rarely needs the biggest model.
- Limit history. Summarise older turns instead of resending everything.
- Set usage limits or tiers. A product decision, not just an engineering one.
Each one can affect quality, so I'd rerun the evaluation set after any change. The AI evaluation checklist covers what that should include.
Put the number where decisions get made
I'd put estimated cost per active user on the same dashboard as adoption and quality, and review it monthly. Not because cost should drive every decision, but because it changes which decisions are sensible.
A feature that costs a little per user might be fine to include in every plan. One that costs a lot might belong in a higher tier, or need a usage limit, or need redesigning. You can only have that conversation if someone has done the arithmetic.
Provider prices change often and usually go down, so revisit the model every few months. An idea that was too expensive last year might be fine now.