Most of the cost threads here end up in the same place: someone swaps to a cheaper model and the bill barely moves. Sharing the breakdown that finally made it click for us, because the savings are almost never in the model choice.
Input tokens usually account for 60 to 75 percent of an LLM bill, and a large share of them are identical between calls. Your system prompt ships with every request. Your tool definitions ship with every call, including the tools that will never fire on that request. The conversation history is resent in full on every follow up so the model can rediscover what it already saw.
Run it on a concrete case. A support assistant handling 10,000 conversations a day at 8 turns each: 2,000 tokens of system prompt, 1,500 of tool definitions, 1,500 of retrieved chunks, and by turn 8 roughly 4,000 tokens of accumulated history. That averages near 50,000 input tokens per conversation, about USD 45,000 a month in input alone at USD 3 per million, before output tokens push the total past USD 60,000.
Prompt Caching Is The Cheapest Win
If the front of your prompt is byte identical between requests, providers can cache that prefix and charge a fraction for it on hits. It needs no architectural change, only discipline: stable content first, anything variable last. The usual way teams break this is interpolating a timestamp or a user id near the top of the system prompt, which quietly kills every cache hit.
Routing And Memory Change The Slope
Model routing is the second lever. Classify the request before you spend on it and send the short, repetitive majority to a smaller model. Most production traffic never needed a frontier model.
The third lever changes the shape of the curve rather than the constant. If the model needs a preference the user stated on turn two, storing that fact and retrieving it costs far less than resending the whole transcript every turn so it can be rediscovered. Context stops growing linearly with session length. The full anatomy of the bill and each strategy is written up in this guide to AI cost optimization with the numbers behind it.
Measure Before You Optimize
Log input tokens, output tokens and model per request, grouped by feature. Almost every team that instruments this finds a single feature consuming most of the budget, and it is rarely the one they would have guessed.
