A practical LLM evaluation framework helps you compare prompts and models with evidence instead of intuition. This guide provides a reusable structure for test cases, evaluation metrics, human review, regression testing, and update decisions.
Overview
LLM evaluation is the process of checking whether an AI system produces outputs that are useful, accurate, safe, consistent, and appropriate for its intended task. It applies to a single prompt, a complete AI feature, or a production workflow that combines retrieval, tools, routing, and structured output.
A good evaluation system does more than assign a score to a model. It creates a repeatable way to answer practical questions: Did a prompt change improve the result? Does a cheaper model meet the same quality threshold? What happens when user inputs are ambiguous? Has output behavior changed after a model, retrieval source, or application update?
Start with the task rather than with a favorite metric. A summarizer, classifier, support assistant, and JSON-producing agent require different tests. Define what a successful output must do, what it must never do, and which failures matter most to users or operators. Your evaluation set should then represent those conditions, including ordinary inputs, difficult examples, edge cases, and known failure modes.
For a broader set of scenarios, use the AI QA test case library as a planning reference. The goal is not to test every possible input. It is to create a curated set that is small enough to run regularly and rich enough to expose meaningful regressions.
Template structure
A reusable LLM evaluation framework can be organized into six connected parts.
1. Task definition
Record the feature name, intended user, input format, expected output format, and decision the model is meant to support. State whether the evaluation concerns a prompt, model, retrieval pipeline, tool call, or the complete application. This prevents a narrow prompt test from being mistaken for an end-to-end quality assessment.
2. Versioned test cases
Give every case a stable identifier and store the input, relevant context, expected properties, risk category, and priority. A test case might include an ideal answer, a set of acceptable answers, or a rubric describing required and prohibited behavior. Keep the original input unchanged when possible so that results remain comparable over time.
Useful test categories include normal usage, incomplete instructions, conflicting information, unsupported requests, sensitive content, long context, multilingual input, formatting errors, and adversarial attempts to override instructions. Add cases based on real failures, not only hypothetical examples.
3. Evaluation metrics
Use a small collection of metrics that reflects the task. Common dimensions include factual correctness, relevance, completeness, instruction following, clarity, groundedness, refusal behavior, latency, cost, and structured-output validity. Not every dimension needs a numerical score. Some requirements are better represented as pass-or-fail checks, such as valid JSON, presence of required fields, or refusal of a disallowed operation.
4. Scoring method
Define how each metric is assessed before comparing outputs. A simple rubric might use a zero-to-two scale: zero for a failure, one for a partial result, and two for a satisfactory result. Include short scoring descriptions so different reviewers apply the rubric consistently. If you use an automated evaluator or an LLM-as-a-judge, validate it against human-reviewed examples and monitor disagreement. The LLM-as-a-judge guide covers situations where this approach is useful and where caution is warranted.
5. Comparison and decision rules
Compare a candidate prompt or model with a named baseline. Report results by metric and by test category, not only as one blended score. Set decision rules in advance, such as requiring no critical safety failures, a minimum structured-output pass rate, or an improvement on a priority category without unacceptable cost or latency changes.
6. Review record
Store the test-set version, prompt version, model identifier, relevant settings, retrieval configuration, date, evaluator version, results, and reviewer notes. This record turns an experiment into an auditable development artifact and makes later regressions easier to investigate.
A compact test-case schema might look like this:
{
"id": "support-014",
"category": "ambiguous_request",
"input": "Can you cancel it?",
"context": "The user has two active subscriptions.",
"must": ["ask which subscription"],
"must_not": ["claim cancellation completed"],
"priority": "high"
}How to customize
Customize the framework according to risk, volume, and the type of output being evaluated.
Choose metrics that match the failure
For classification, measure label correctness, confusion between important classes, and confidence calibration where confidence affects downstream actions. For summarization, assess factual consistency, coverage of important points, and unsupported additions. The guides to classification evaluation and summarization quality provide task-specific starting points.
For retrieval-augmented generation, separate retrieval quality from answer quality. Test whether the correct sources are retrieved, whether the response uses those sources, and whether it clearly handles missing evidence. A fluent answer can still fail if its supporting context is incomplete or irrelevant.
Separate automated checks from human review
Automated checks are efficient for repeatable properties: schema validity, required fields, citation presence, prohibited phrases, exact labels, latency, and token or cost thresholds. Human review is more useful for nuanced qualities such as usefulness, tone, reasoning transparency, and whether an answer adequately addresses the user's intent.
Use human review selectively. Review all critical failures, a representative sample of passing outputs, and cases where automated checks disagree. Track reviewer agreement and refine ambiguous rubric language rather than treating disagreement as noise.
Test the complete AI workflow
Prompt testing alone can miss failures caused by retrieval, preprocessing, tool execution, routing, truncation, or post-processing. Include end-to-end cases for production paths. If your system routes requests between models, evaluate both the routing decision and the resulting answer; the model routing guide offers a useful way to frame those comparisons.
For applications that return JSON or call functions, test syntax, schema compliance, field types, missing values, extra fields, and behavior when the model cannot complete the operation. See structured output reliability testing for a focused checklist.
Keep scores interpretable
A dashboard should help a team decide what to do next. Avoid combining unrelated metrics into a single quality number unless the weighting is explicit and reviewed. Show pass rates, failure counts, category-level results, and examples of representative failures. A scorecard can be useful when it supports decisions rather than replacing them; see AI evaluation dashboard metrics for ideas on what to track.
Examples
Example: a support-answer prompt
Suppose a support assistant must answer from an approved knowledge base. Build cases for straightforward questions, missing articles, contradictory context, account-specific requests, and attempts to obtain information the assistant cannot verify. Score groundedness, relevance, completeness, appropriate escalation, and refusal to invent account actions. A release should not be approved solely because average helpfulness improved if the system also began making unsupported claims.
Example: a structured extraction prompt
For an invoice extraction workflow, create cases with complete invoices, missing fields, unusual date formats, multiple line items, poor image quality, and conflicting totals. Use automated checks for valid JSON, schema compliance, data types, and arithmetic consistency. Have reviewers assess whether extracted values match the source and whether uncertainty is represented rather than hidden.
Example: prompt regression testing
Keep a baseline prompt and run it against the same versioned test set whenever a candidate prompt is proposed. Compare not only the overall pass rate but also high-priority failures. A prompt that improves concise answers while weakening instruction following in edge cases may need a targeted revision rather than immediate adoption. The production prompt review checklist can complement these tests.
For each failed case, record the observed output, failure category, likely cause, and proposed fix. This creates a feedback loop for prompt optimization: improve the instruction, add or revise a test, rerun the baseline, and confirm that the change does not introduce a new regression.
When to update
Revisit the evaluation framework whenever an input to the AI workflow changes. Important triggers include a new model or model version, prompt edits, changes to system instructions, retrieval-index updates, new tools, altered routing logic, output-schema changes, preprocessing changes, and adjustments to safety or escalation rules.
Update the test set after a production incident, a newly observed user pattern, or a human-review finding that is not represented by an existing case. Do not delete old cases simply because the system now passes them; they provide regression coverage. Add a category or priority label when the risk profile changes, and document why the change was made.
Review metrics and rubrics periodically as the product evolves. A metric that mattered during prototyping may be less important in production, while latency, reliability, or escalation accuracy may become more significant. If evaluation results are difficult to interpret, simplify the rubric and preserve concrete examples of pass and fail behavior.
Use this maintenance routine after each meaningful change:
- Freeze the candidate configuration and record its versions.
- Run the full regression set, including critical safety and formatting checks.
- Compare results with the current baseline by category.
- Review all critical failures and a sample of automated passes.
- Decide whether to ship, revise, or expand the test set.
- Record the decision, evidence, and follow-up owner.
Finally, watch for behavior changes that appear without a deliberate prompt edit. Differences in model availability, retrieved content, tool responses, or upstream data can alter outputs. The AI output drift guide provides a framework for detecting and responding to those changes. A maintained evaluation set is not a one-time benchmark; it is the working test suite for your production AI workflow.
