Switching an LLM provider is not only an API rewrite. It is a controlled change to authentication, request behavior, output quality, latency, cost, safety, and operational recovery. This checklist gives you a repeatable way to compare providers, replay representative traffic, estimate total impact, and decide whether to use shadow traffic, a canary release, dual running, or a tested rollback.
Overview
An LLM provider migration is successful when the replacement works for the application’s real workload, not merely when a sample request returns a plausible answer. A provider can appear compatible at the endpoint level while differing in tokenization, streaming events, structured-output enforcement, tool-call syntax, refusal behavior, rate limits, or error handling.
Separate the migration into four questions:
- Can the application connect? Verify authentication, endpoints, SDK behavior, regions, timeouts, retries, and quotas.
- Can the model perform the required work? Compare task quality, instruction following, tool use, structured outputs, safety behavior, and failure modes.
- Can the business operate it? Estimate cost, latency, throughput, observability, data handling, support requirements, and incident procedures.
- Can the team reverse the change? Keep the existing path deployable and define measurable rollback triggers before cutover.
Begin with an inventory rather than changing code immediately. Record every model, endpoint, prompt version, parameter, tool schema, output schema, retry rule, token limit, and downstream assumption. Include secondary paths such as summarization, classification, moderation, batch jobs, evaluation scripts, and internal tools. A migration can fail because an infrequently used workflow was omitted from the test set.
For quality measurement, create a representative evaluation set that includes routine, ambiguous, long-context, adversarial, and known-failure cases. The LLM evaluation dataset design guide can help structure that set. Keep a human-review path for consequential outputs; automated judges are useful for scale but should not be the only acceptance check.
How to estimate
Use a replay-based comparison. Capture sanitized production requests or construct an approved test set, then send equivalent cases to the current and candidate providers under controlled conditions. Preserve the same user intent, relevant context, tool definitions, output schema, and application-side post-processing. Do not assume that identical temperature or sampling settings create equivalent behavior across providers.
A useful request-replay record contains:
{
"case_id": "support-0042",
"workflow": "ticket_triage",
"prompt_version": "triage-v7",
"input_class": "ambiguous",
"messages": "sanitized conversation payload",
"tools": ["lookup_customer", "classify_issue"],
"expected_output": "schema or reviewer rubric",
"current_result": "stored response and metadata",
"candidate_result": "stored response and metadata",
"metrics": {
"valid_schema": true,
"task_score": null,
"input_tokens": null,
"output_tokens": null,
"time_to_first_token_ms": null,
"total_latency_ms": null,
"retry_count": null
}
}Compare results by workflow, not only by an overall average. Track at least:
- Compatibility: successful requests, HTTP and SDK errors, timeout rate, stream completion, tool-call parsing, and schema validity.
- Quality: task accuracy, groundedness where relevant, instruction adherence, completeness, unwanted verbosity, and human preference.
- Reliability: latency distributions, retry frequency, rate-limit responses, transient failures, and behavior under concurrency.
- Cost: input and output token volume, fixed platform charges if applicable, retries, fallback calls, and supporting infrastructure.
- Safety: prompt-injection resistance, sensitive-data handling, refusal consistency, unsafe transformations, and leakage through tool arguments or logs.
For a simple cost estimate, define:
Estimated monthly cost = input tokens ÷ billing unit × input rate + output tokens ÷ billing unit × output rate + request-related platform costs + retry and fallback costs.
Use the candidate provider’s current published pricing or contract terms as inputs rather than copying assumptions from an old comparison. Model at least three traffic levels: expected, high, and failure-heavy. A cheaper nominal request may not remain cheaper if it produces more output, requires more retries, or needs a second model to repair invalid JSON.
For a broader decision, calculate a weighted migration score. For example, assign weights to compatibility, quality, latency, cost, safety, and operational fit, then score each provider against the same rubric. The score is a decision aid, not a substitute for hard gates. A candidate that fails structured-output validity or data-handling requirements should not pass because it performs well on cost.
Inputs and assumptions
Document the inputs before testing so the comparison can be repeated after a model alias, prompt, traffic pattern, or price changes. At minimum, capture:
| Area | Inputs to record | Acceptance question |
|---|---|---|
| API and SDK | Endpoint, authentication, SDK version, regions, timeouts, retries | Can the application connect and recover predictably? |
| Conversation behavior | Message roles, context limits, system instructions, tokenization assumptions | Does the same context fit and retain the required meaning? |
| Streaming | Event format, partial text, finish signals, disconnect handling | Can the user interface and persistence layer consume the stream? |
| Structured output | Schema syntax, strictness, refusal representation, repair logic | Are outputs valid without unsafe or excessive repair? |
| Tools | Function names, argument schemas, parallel calls, tool-result messages | Do tool calls remain safe, parseable, and complete? |
| Operations | Rate limits, concurrency, quotas, logs, traces, alerts | Can the team see and control failures? |
| Data and safety | Approved data classes, retention assumptions, regional needs, redaction | Does the deployment meet internal requirements? |
Keep provider-specific code behind an adapter where practical. Normalize your internal request and response objects, then isolate differences in authentication, message formatting, streaming events, tool calls, usage metadata, and error mapping. This reduces future switching cost, but it does not make providers interchangeable. The adapter should expose meaningful differences instead of hiding them behind a misleading lowest-common-denominator interface.
For structured outputs, test valid responses, missing fields, extra fields, wrong types, refusal paths, truncated responses, and malformed tool arguments. See the structured output evaluation guide for a focused test plan. For prompt changes made during migration, run a prompt regression suite rather than approving a few hand-picked examples; the prompt regression testing workflow provides a practical CI pattern.
Worked examples
Example 1: Cost and quality comparison
Suppose a team has three workflows: support classification, answer drafting, and document extraction. It records monthly input and output token volumes, then replays a fixed evaluation set against the current and candidate providers. The candidate shows lower estimated request cost, but extraction has a higher invalid-schema rate and answer drafting has longer tail latency.
The team should not average these results into one number. It can set workflow-specific gates: classification must preserve the agreed quality band, extraction must meet the schema-validity threshold, and drafting must remain within the product’s latency budget. It can then estimate repair and fallback costs for extraction before deciding whether the candidate is actually less expensive.
For a deeper replay method covering model and prompt changes, use the LLM cost-quality optimization workflow. Record the assumptions beside every estimate: traffic volume, average context size, output ceiling, retry rate, fallback rate, and the pricing or contract version used.
Example 2: Choosing a cutover method
| Method | Use when | Primary control |
|---|---|---|
| Shadow traffic | You need realistic comparison without changing user-visible responses. | Candidate receives copied or replayed requests; its output is not served. |
| Canary release | The candidate has passed offline tests and limited live exposure is acceptable. | Route a defined, observable slice and expand only if gates hold. |
| Dual running | The workflow is critical or rollback confidence is more important than simplicity. | Keep both paths available while comparing live outcomes and operating cost. |
Before a canary, define thresholds for error rate, schema failures, quality complaints, latency, cost per completed task, and safety incidents. Define the observation window and who can stop the rollout. A rollback should be a configuration change or deployable release, not an emergency code rewrite.
Migration sign-off checklist
- All production workflows and model dependencies are inventoried.
- Sanitized replay cases cover normal, edge, long-context, and adversarial inputs.
- Authentication, quotas, rate limits, retries, streaming, and timeouts are tested.
- Tool calls and structured outputs pass parsing and safety checks.
- Quality is reviewed by workflow, with human checks for consequential cases.
- Cost estimates include token volume, retries, fallbacks, repairs, and support overhead.
- Logs, metrics, traces, alerts, and provider request identifiers are available.
- Rollback configuration is tested and the owner of the decision is named.
When to recalculate
Recalculate the migration decision whenever an input that affects compatibility, quality, cost, or recovery changes. That includes provider pricing or contract terms, model retirement or alias changes, API and SDK updates, new rate limits, regional deployment, retention or data-handling changes, significant prompt revisions, new tools, longer context, higher traffic, or a change in the application’s quality threshold.
Also recalculate after incidents. A production failure may reveal that the original test set omitted a timeout pattern, tool failure, prompt-injection attempt, malformed document, or high-concurrency condition. Add the incident to the replay suite, compare the current and fallback paths, and update the rollback trigger. The LLM observability guide outlines the measurements needed to detect this kind of drift.
Maintain a dated migration record containing the provider and model identifiers, API version, adapter version, prompt versions, evaluation-set version, pricing assumptions, measured results, and approval decision. Review it before a contract renewal, regional launch, major model change, or planned fallback activation. A provider migration is safest when it is treated as a repeatable AI development workflow: measure first, expose differences explicitly, release gradually, and keep a tested path back.