Reducing LLM spend without lowering answer quality requires more than choosing the cheapest model. This replay-based workflow shows how to compare models, prompts, context sizes, and generation settings against the same representative requests, then make a decision using cost per task, quality thresholds, latency, and operational risk.
Overview
LLM cost optimization is a model evaluation problem as much as a billing problem. A cheaper model may require longer prompts, more retries, stricter post-processing, or human review. A larger model may produce better first-pass answers but add cost that does not improve the outcome users actually need. The useful comparison is therefore not price per token alone. It is the cost of completing an acceptable task.
A replay-based workflow creates a controlled comparison. You collect a representative sample of historical or test requests, run each request through several candidate configurations, measure token use and operational behavior, and score the outputs against predefined acceptance criteria. The configuration that wins is not necessarily the lowest-cost option. It is the one that meets the required quality and safety bar at an acceptable total cost.
This approach also makes prompt optimization more disciplined. Instead of changing a prompt and relying on a few anecdotal examples, you can compare the old and new versions on the same dataset. The resulting evaluation set becomes a reusable asset for model upgrades, provider changes, and production AI workflow reviews.
How to estimate
Start by defining the unit you want to optimize. For a support assistant, it might be one resolved conversation. For extraction, it might be one document processed. For a summarizer, it might be one summary that passes review. Calling this unit a task prevents the analysis from stopping at token pricing.
- Build a replay set. Select representative requests covering common cases, long inputs, ambiguous instructions, edge cases, and known failures. Remove or protect sensitive data before replaying production examples. A small but varied set is more useful than a large set made up only of easy requests.
- Freeze the comparison conditions. Keep the user input, system instructions, retrieved context, output schema, and evaluation rules consistent where possible. Record the model identifier, prompt version, context length, generation settings, retry policy, and date of the run.
- Run candidate configurations. Compare the current setup with alternatives such as a smaller model, a shorter prompt, reduced retrieved context, a different output limit, or a fallback route. Run enough repetitions to account for output variability when the application is nondeterministic.
- Measure cost and behavior. Capture input tokens, output tokens, estimated provider charges, latency, errors, retries, truncation, schema failures, and safety flags. Use the provider's current pricing and token-counting method for the deployment being tested; do not transfer rates from an old model or plan without checking them.
- Score quality against acceptance criteria. Use task-specific checks such as factual accuracy, required-field completeness, citation or evidence coverage, instruction following, tone, safety, and format validity. Automated checks can screen every result, while human review or an LLM judge can assess dimensions that require interpretation.
- Apply the decision rule. Reject configurations that miss a non-negotiable threshold, even when they are inexpensive. Among the remaining candidates, compare cost per accepted task, latency, reliability, and implementation complexity.
A practical cost-per-task formula is:
Cost per task = model charges + retries + routing overhead + review or correction cost
If review or correction cost is difficult to price, track it as a separate rate. For example, report both “API cost per task” and “accepted-task rate.” This makes it clear whether savings come from genuine efficiency or from allowing more failures through.
Inputs and assumptions
Keep a worksheet for every candidate configuration. The following fields are usually enough to reproduce a comparison:
| Input | What to record | Why it matters |
|---|---|---|
| Request volume | Tasks per day or billing period | Converts unit economics into an operating estimate |
| Input tokens | Average and distribution, not just the mean | Long-context requests can dominate spend |
| Output tokens | Average, maximum, and truncation rate | Verbose answers may increase cost without improving quality |
| Pricing inputs | Current input, output, cached, or batch rates where applicable | Pricing plans and billing features can change |
| Quality score | Pass rate and scores by important criterion | Prevents cost savings from hiding regressions |
| Operational metrics | Latency, errors, retries, and timeouts | Failures create indirect cost and user friction |
| Context policy | Retrieved documents, truncation rules, and maximum context | Prompt and RAG design strongly influence token use |
Define acceptance thresholds before looking at the results. A template might state: “At least 95% of outputs must pass schema validation; factual accuracy must not fall below the current baseline; safety-critical cases require zero unresolved violations; and p95 latency must remain within the product target.” The exact thresholds belong to the task and its risk level. A creative drafting tool and a financial data extraction workflow should not use the same standard.
Separate hard constraints from preference scores. A hard constraint might be valid JSON or the presence of every required field. A preference score might measure helpfulness on a five-point scale. This prevents a high general score from compensating for a critical failure.
Worked examples
Assume a team is testing three configurations for a document extraction task. The figures below are illustrative placeholders, not current provider prices:
| Configuration | Estimated API cost per task | Accepted-task rate | Average latency | Decision view |
|---|---|---|---|---|
| Baseline model and prompt | 0.010 units | 96% | 2.4 seconds | Quality reference |
| Smaller model, same prompt | 0.004 units | 91% | 1.6 seconds | Fails the 95% threshold |
| Smaller model, shorter prompt, constrained output | 0.005 units | 95% | 1.7 seconds | Potential replacement |
If the application processes 100,000 tasks in a period, the baseline's estimated API spend is 1,000 units using the illustrative figures. The third configuration is 500 units before accounting for retries, review, or other overhead. That apparent saving is meaningful only if the 95% acceptance rate holds across important subgroups, including long documents and unusual fields.
Now test routing. A fast, lower-cost model can handle routine requests, while a stronger model receives cases that match a defined risk or complexity rule. Evaluate the router itself: measure misrouted requests, fallback frequency, and the quality of tasks sent to each path. A routing policy that sends too many difficult cases to the fallback model may cost nearly as much as using the stronger model for every request.
Also test caching and batching separately from model quality. Caching repeated, identical work can reduce duplicate calls, but it requires a clear invalidation policy and careful handling of user-specific data. Batching may improve economics for asynchronous workloads, but it may not suit interactive requests that require immediate responses. These are workflow decisions, not substitutes for evaluation.
When to recalculate
Re-run the comparison whenever an input to the decision changes. At minimum, revisit it after a provider pricing update, model replacement, prompt revision, context-window change, new batch or caching option, major traffic shift, or change in the application's acceptance criteria. A monthly cost review is a useful baseline for production systems, but high-risk or rapidly changing workflows may need more frequent checks.
Use monitoring to decide whether a full replay is necessary. Track cost per task, token distribution, acceptance rate, latency, retry rate, fallback rate, and failure categories. A rising average cost may be caused by longer user inputs rather than a price change. A falling quality score may indicate prompt drift, retrieval changes, or a new request mix. Group metrics by task type so that easy cases do not conceal failures in a critical segment.
Before approving a change, complete this checklist:
- Confirm the pricing, token-counting method, model limits, and billing assumptions.
- Replay the same versioned dataset against the current and candidate configurations.
- Compare accepted-task cost, not token price alone.
- Inspect failures by category, risk level, and input length.
- Verify structured output, safety checks, retries, fallbacks, and logging.
- Record the decision, thresholds, dataset version, and rollback condition.
- Schedule a follow-up review after the change reaches representative traffic.
Keep the replay set and worksheet in version control alongside prompt and application changes. That turns LLM cost optimization into a repeatable AI development workflow rather than a one-time search for a cheaper model. For deeper implementation, connect the process to LLM observability, prompt regression testing, and a representative LLM evaluation dataset. Recalculate when prices, traffic, prompts, or benchmarks move, and treat every saving as provisional until quality remains within the agreed limits.