LLM Cost Optimisation: Controlling Token and Inference Costs in AI Products
Practical ways to control LLM costs in AI products: measuring cost per task, prompt and context design, caching, model routing, batching and usage limits.

In this article
- Why AI Costs Surprise Teams
- Step 1: Measure Cost Per Task
- Step 2: Trim the Context
- Step 3: Cache
- Step 4: Route to the Right Model
- Step 5: Batch and Schedule
- Step 6: Control Agents and Loops
- Step 7: Set Limits and Price Sensibly
- Self-Hosting Open Models
- Example Savings Plan
- Keep Quality Visible
- Recommendations
- How Webespire Consulting Can Help
- Frequently Asked Questions
- Conclusion
- Further Reading
LLM cost optimisation means reducing what you spend on language model usage per useful outcome without lowering quality below an agreed level. The biggest levers are usually measuring cost per task, sending less unnecessary text to the model, caching repeated work, routing simple requests to smaller models, using batch processing for non-urgent jobs, and setting sensible usage limits. Every change should be checked against an evaluation set so savings do not quietly damage results.
Why AI Costs Surprise Teams
Model pricing is usually per token, separately for input and output. Costs grow with prompt length, conversation history, retrieved documents, tool calls and agent loops. A feature that cost little in testing can become expensive when real users paste long documents, hold long conversations or trigger multi-step agents. Because pricing changes frequently, build your cost model from your provider's current price list and real usage data rather than fixed assumptions.
Step 1: Measure Cost Per Task
Log tokens in and out, model used, latency and outcome for every request, tagged by feature and customer. Then calculate cost per completed task: per answered question, per processed document, per generated report. This shows which features and customers drive cost and gives a baseline for improvements.
Step 2: Trim the Context
- Shorter system prompts. Remove repeated or obsolete instructions.
- Fewer, better retrieved passages. Improve retrieval and re-ranking so you send three relevant passages instead of fifteen loosely related ones.
- Summarise long conversations rather than resending the full history every turn.
- Avoid sending whole documents when only a section is needed.
Context trimming often improves quality as well as cost, because the model sees less noise.
Step 3: Cache
- Prompt caching. Several providers discount repeated prompt prefixes such as long system prompts or shared documents. Structure prompts so stable content comes first.
- Response caching. For identical or near-identical questions (common in support), return a stored answer when it is still valid.
- Intermediate results. Cache embeddings, document summaries and extraction results instead of recomputing them.
Step 4: Route to the Right Model
Not every request needs the most capable model. Use a smaller, cheaper model for classification, simple extraction, routing and short answers, and reserve larger models for complex reasoning. A router can be a simple rule ("documents under two pages go to the small model") or a lightweight classifier. Validate routing with evaluation results on each path.
Step 5: Batch and Schedule
Work that does not need an instant answer, such as overnight document processing, report generation or re-indexing, can often use providers' batch interfaces at lower prices or run at off-peak times on self-hosted infrastructure.
Step 6: Control Agents and Loops
Agents that call tools repeatedly can multiply cost. Cap the number of steps, stop when a task is clearly failing, and prefer deterministic code for steps that do not need a model. Log each step so expensive patterns are visible.
Step 7: Set Limits and Price Sensibly
- Per-user and per-account usage limits, with clear messages when reached.
- Rate limits to stop abuse.
- Pricing that reflects cost drivers: usage-based components or plan limits for heavy AI features. Our article on SaaS subscription billing architecture explains how to model usage.
Self-Hosting Open Models
Running open-weight models on your own infrastructure can lower marginal cost at high, steady volume and keep data in your environment. It adds operational work: GPU capacity, scaling, monitoring and model updates. Compare total cost, including engineering time and idle capacity, against API pricing before switching.
Example Savings Plan
| Change | Effect on cost | Quality risk | Check |
|---|---|---|---|
| Remove unused instructions from system prompt | Lower input tokens on every call | Low | Regression evaluation |
| Retrieve 4 passages instead of 12, with re-ranking | Lower input tokens | Medium | Groundedness and correctness scores |
| Route classification to a smaller model | Lower price per call | Medium | Accuracy on classification set |
| Cache answers to frequent questions | Fewer calls | Low if invalidated on content change | Freshness checks |
| Batch overnight extraction | Lower unit price where offered | None for latency-tolerant jobs | Job monitoring |
Keep Quality Visible
Run your LLM evaluation suite before and after each change and track quality alongside cost on the same dashboard. A cheaper feature that users stop trusting is not a saving.
Recommendations
- Log tokens, model and outcome for every request; compute cost per task.
- Trim prompts and retrieved context first.
- Add caching and smaller-model routing where tests allow.
- Use batch processing for non-urgent work.
- Cap agent steps, set usage limits and align pricing with costs.
How Webespire Consulting Can Help
Our AI solutions team profiles AI usage, implements caching and routing, and builds the monitoring that keeps cost and quality visible. For architecture choices that affect cost, read AI SaaS development and RAG vs fine-tuning.
Frequently Asked Questions
What usually drives LLM costs the most?
Long inputs: conversation history and retrieved documents sent on every call, plus agent loops with many steps.
Will smaller models reduce quality?
For simple tasks often not noticeably; for complex reasoning they may. Test each route against your evaluation set.
Is self-hosting cheaper than using an API?
Sometimes at high, steady volume. Include infrastructure, operations and idle capacity in the comparison.
How do we stop one customer from running up costs?
Per-account limits, rate limits and usage-based pricing for heavy features.
How often should we review AI costs?
Monthly at least, and after any change to prompts, models or traffic patterns.
Conclusion
LLM costs are manageable when they are measured per task and treated as an engineering metric. Trim context, cache, route and batch, set limits, and prove every change with evaluations. The result is AI features whose economics improve as usage grows.
Further Reading
Want to bring AI running costs under control? Email info@webespire.com or visit webespire.com.



