The bill arrives
Most AI teams don't think about cost until the invoice arrives. By then the patterns are baked in: every call hits the flagship model, every request ships the full conversation history, and nobody checked whether any of those tokens were actually necessary.
In practice, three patterns account for roughly 80% of LLM overspend: not caching repeated prompt prefixes, always routing to the most expensive model, and making synchronous API calls for tasks that don't need a real-time response. Each has a direct fix. Together, they typically halve the bill — often more.
The 60-second cost model
Before optimizing, you need a mental model of what you're paying for. Every LLM API charges by tokens: separately for input tokens and output tokens, with output usually costing 3–5× more per token than input.
A rough 2026 reference point for major frontier APIs:
- Flagship model (e.g. Claude Opus, GPT-4o): ~$15/M input, ~$75/M output
- Mid-tier model (e.g. Claude Sonnet, GPT-4o-mini): ~$3/M input, ~$15/M output
- Fast/cheap model (e.g. Claude Haiku, GPT-3.5-tier): ~$0.25/M input, ~$1.25/M output
If a task doesn't require frontier-level reasoning, every call you make to the flagship model is a 5–60× surcharge over what the same task would cost on a smaller model.
Measure your token distribution first. In most products, 60–80% of calls are simple tasks: classification, extraction, reformatting, short-form generation with tight constraints. Those are mid-tier or fast-model tasks.
Pattern 1: Prompt caching
If your prompt shares a stable prefix — a system prompt, a document, a set of instructions — across many calls, prompt caching is the highest-leverage optimization available.
Both Anthropic and OpenAI support it. You mark a prefix as cacheable; on first use it's written to cache (billed at normal rates plus a small write surcharge), and on subsequent calls within the cache window, those tokens are read at 90% discount. A system prompt you send 1000 times per day goes from costing $0.15/day to $0.016/day.
The mechanics vary by provider, but the principle is the same: separate the parts of your prompt that change from the parts that don't, and cache the stable prefix.
Common candidates for caching:
- System prompts with tool definitions (often 500–2000 tokens)
- Base documents for a RAG-style feature (if the corpus is small and stable)
- Few-shot examples that don't change per request
- Long policy or persona blobs
The 2025 Anthropic prompt caching paper showed cache hit rates of 70–90% in production workloads where teams had explicitly separated system and user turns. In those workloads, effective input cost dropped by 60–75%.
Pattern 2: Model tiering
Route different tasks to different model tiers. This sounds obvious; very few teams actually do it systematically.
A tiering schema:
| Task type | Suitable tier | Reason |
|---|---|---|
| Intent classification | Fast model | Binary or small-label output |
| JSON extraction (structured template) | Fast model | Pattern matching, not reasoning |
| Summarization, short form | Mid-tier | Coherent prose, no complex reasoning |
| Multi-step reasoning, complex coding | Flagship | Genuinely needs the capability |
| Agentic loops | Flagship | Reliability > cost per step |
The key discipline is building a test harness before tiering down. Run 50–100 representative cases through both the expensive model and the candidate cheaper model. If quality is within your acceptable range — often it is for extraction and classification tasks — you've earned a 5–60× cost reduction on that task class.
Tools worth knowing: LiteLLM provides a unified interface for routing to different providers; you can add model-routing middleware without rewriting call sites.
Pattern 3: Async batching
Not every LLM call needs a real-time response. Analytics pipelines, nightly document processing, bulk classification, pre-computed summaries — these are all batch workloads.
Anthropic's Batch API (and equivalent services on other providers) gives a 50% discount on all tokens in exchange for a 24-hour latency window. For tasks running in background jobs, this is free money.
The pattern: tag every API call site in your codebase as either real-time-required or async-eligible. Real-time goes through the normal API. Async-eligible accumulates in a queue and flushes to the Batch API on a schedule.
Even modest async volumes add up fast. If you're running 500k input tokens/day on batch-eligible tasks at flagship model rates, the 50% discount saves ~$3.75/day — $1,370/year — for zero quality change.
Pattern 4: Context pruning
Context bloat is the cost leak teams notice last. In a multi-turn conversation or agent loop, the context grows with every turn. If you're retransmitting the full history on every API call, you're paying for tokens the model already processed.
Two fixes:
Summarize old turns. When a conversation exceeds N turns or K tokens, run a summarization step on the oldest N-3 turns, replace them with the summary, and continue. The model retains the substance; you cut context size by 50–70%.
Prune tool observations. In agentic loops, raw tool responses can be verbose — full HTML pages, large JSON blobs. Post-process tool outputs before returning them to the model: extract the relevant fields, trim to relevant sections, strip boilerplate. A tool response that was 3000 tokens becomes 200 tokens with no information loss.
The monitoring layer you need first
You can't optimize what you can't measure. Before any of the above patterns are useful, instrument:
- Token counts per call (input and output separately)
- Model used per call
- Task type or route (so you can see which routes are expensive)
- Cache hit/miss (once you add caching)
A simple structured log to your observability layer is enough. Run a weekly cost-by-route breakdown. The two or three most expensive routes are where all the optimization leverage lives.
The tradeoff you should NOT make
Cost optimization becomes counterproductive when it degrades quality on tasks that matter. The wrong sequence: cut to the cheapest model, ship, discover regressions in production, roll back under pressure.
The right sequence: instrument, identify the expensive + low-value tasks, run a quality evaluation before tiering down, ship with a test harness that alerts on quality regression.
Cost and quality are not in fundamental tension — they're in tension only when you optimize without measuring. The teams running the leanest LLM infrastructure in 2026 are also running the most rigorous eval discipline, because eval is what makes confident cost reductions possible.
The [Deployment & MLOps course](https://nextgenailearn.com/paths/deployment-mlops) covers the full production cost-optimization stack: model serving, cost dashboards, tiering strategies, and the observability layer that makes all of it safe to deploy. If you're preparing for an MLOps or Applied GenAI Engineer role, the [MLOps Fundamentals cert practice pack on CertQuests](https://certquests.com/packs/mlops-fundamentals) includes a dedicated cost-optimization question bank built around these exact patterns.