A reliable RAG evaluation process separates retrieval problems from generation problems, measures answer quality against evidence, and turns failures into reusable test cases. This checklist shows how to build a practical RAG test dataset, score retrieval relevance, inspect groundedness and citations, and decide when your evaluation suite needs to change.
Overview
Retrieval-augmented generation (RAG) systems usually have at least two connected stages: a retriever selects passages from a knowledge base, and a language model uses those passages to produce an answer. An answer can fail because the right information was never retrieved, because the model misunderstood relevant context, or because the source content itself is incomplete or outdated. Evaluating only the final response makes these causes difficult to distinguish.
A useful RAG evaluation therefore measures both retrieval quality and answer quality. Retrieval metrics ask whether the system found the right evidence. Generation metrics ask whether the answer is supported, complete, relevant, and accurately cited. Human review remains valuable for nuanced cases, but a structured dataset and consistent rubric make that review faster and more repeatable.
For broader guidance on test cases, metrics, and evaluation workflows, see the LLM evaluation framework. RAG testing can then be added as a focused layer within that wider model evaluation process.
A practical test-case format
Store each evaluation example with enough information to reproduce and diagnose the result. A compact record might include:
- Question: the user query as it would appear in the application.
- Expected answer: a reference answer, key facts, or an acceptable answer range.
- Relevant source: the document, section, or passage that should support the response.
- Metadata: product area, document version, access scope, language, and difficulty.
- Retrieved chunks: the passages returned by the retriever, including rank.
- Generated answer: the final response and citations shown to the user.
- Review labels: retrieval relevance, groundedness, completeness, citation accuracy, and overall outcome.
Include both ordinary questions and difficult cases: ambiguous wording, multi-step questions, conflicting documents, missing information, permission-restricted content, and questions the system should decline to answer. A dataset that contains only easy questions can make a weak RAG system appear dependable.
Checklist by scenario
When measuring retrieval relevance
- Define what counts as relevant before reviewing results. A passage may be directly answer-bearing, useful background, or irrelevant.
- Record the expected evidence for each question. Where several passages are acceptable, list them as alternatives rather than forcing one exact match.
- Check whether relevant content appears in the top results, not merely somewhere in a large result set.
- Review ranking, not just retrieval. Two systems may return the same documents while placing the useful passage at different positions.
- Test queries that use synonyms, abbreviations, misspellings, natural language, and terminology unfamiliar to the document authors.
Common retrieval metrics include precision at k, which considers how many of the top-k results are relevant, and recall at k, which considers whether the needed evidence appears within the top-k results. Mean reciprocal rank can help assess how early the first relevant result appears. The exact metric matters less than applying it consistently to a clearly labeled dataset.
When measuring answer quality
- Check groundedness: are the answer's factual claims supported by the retrieved context?
- Check completeness: does the response cover the important parts of the question, or does it omit a required condition?
- Check relevance: does it answer the user's question directly without unnecessary material?
- Check factual accuracy: does the response preserve numbers, names, conditions, and scope from the source?
- Check citation accuracy: do cited sources actually support the claims attached to them?
- Check uncertainty behavior: does the system acknowledge when the context is insufficient instead of filling gaps with speculation?
A simple rubric can use a zero-to-two scale for each dimension: zero for a failure, one for a partial result, and two for a satisfactory result. Define the labels with examples. For instance, a grounded answer should not introduce an important claim that is absent from the retrieved context, even if that claim happens to be true elsewhere.
When testing missing or conflicting information
- Add questions whose answers are not present in the knowledge base.
- Verify that the system says it cannot determine the answer when evidence is insufficient.
- Include multiple documents with different publication dates or versions.
- Check whether the response identifies a conflict rather than silently selecting one source.
- Test access boundaries to ensure restricted content is not used or cited for an unauthorized user.
These cases evaluate guardrail behavior as well as retrieval. A useful RAG system is not one that answers every question; it is one that distinguishes supported answers from unsupported ones.
When comparing retrievers, chunking, or prompts
Change one major variable at a time where possible. Keep the question set, corpus snapshot, generation model, and scoring rubric fixed while comparing chunk sizes, overlap, embedding choices, metadata filters, reranking, or prompt versions. This makes an improvement easier to attribute. For prompt and configuration changes, maintain a regression suite and record the version tested; the prompt versioning and regression testing workflow provides a useful companion process.
What to double-check
Separate retrieval failure from generation failure
When an answer is wrong, inspect the retrieved passages before changing the prompt. If the required evidence is absent, investigate indexing, parsing, chunking, query rewriting, filters, or ranking. If the evidence is present but the answer is unsupported or incomplete, investigate the generation prompt, context formatting, model behavior, or citation logic. This separation prevents teams from trying to solve retrieval defects with increasingly complex instructions.
Review the dataset itself
A test dataset is part of the measurement system. Check that questions are representative of real usage, labels are understandable, and reference answers do not encode unnecessary wording preferences. Include examples from different document types and content owners. Keep a held-out set for comparisons when practical, so repeated tuning does not make the system fit only the examples used during development.
Inspect citations at claim level
A response can include a citation while still being poorly supported. Review whether each important claim is traceable to the cited passage, whether the citation points to the correct version, and whether the source location is usable by the reader. If a response combines several facts, determine whether one citation supports all of them or whether separate citations are needed.
Track operational dimensions
Quality is not the only consideration in a production AI workflow. Record latency, retrieval failures, empty-result rates, token usage, and observable error categories alongside answer scores. These measures do not replace relevance or groundedness, but they help explain trade-offs when a change improves quality at the cost of speed or resources. The AI evaluation dashboard metrics guide can help organize a team scorecard.
Common mistakes
- Scoring only the final answer: Without retrieved-context review, teams cannot tell whether the retriever or generator caused the failure.
- Using exact-match answers for open-ended questions: Correct answers can use different wording. Score key facts, support, and completeness instead.
- Relying exclusively on an automated judge: Automated scoring can improve consistency, but ambiguous, high-impact, or disputed cases need human review.
- Ignoring negative examples: Questions with missing evidence are essential for testing abstention and uncertainty behavior.
- Changing several variables at once: Replacing the retriever, chunking strategy, model, and prompt together makes results difficult to interpret.
- Failing to version the corpus: A score is hard to reproduce if documents, permissions, or indexes changed without being recorded.
- Optimizing for averages alone: A strong average can hide serious failures in a particular product area, language, user group, or question type.
When to revisit
Revisit the RAG evaluation suite before seasonal planning cycles, major releases, and changes to the knowledge base or workflow. Update it whenever documents are reorganized, new content types are added, access rules change, the embedding or reranking configuration changes, or the generation model and prompt are replaced.
Use production feedback to add new cases, but preserve older cases so regressions remain visible. Review failing examples by category: missing retrieval, poor ranking, unsupported claim, incomplete answer, citation mismatch, refusal error, and system or access failure. If one category grows, add targeted tests rather than only increasing the dataset size.
Before shipping a change, run this final checklist:
- Freeze and record the corpus, index, configuration, prompt, and model versions.
- Run retrieval metrics on representative and difficult queries.
- Review groundedness, completeness, relevance, and citation accuracy on generated answers.
- Test missing, conflicting, and access-restricted information.
- Compare results with the current baseline, including operational measures.
- Inspect regressions by scenario instead of relying on one aggregate score.
- Save new failures as labeled test cases and assign an owner for follow-up.
For a wider release-readiness review, combine this checklist with the production AI workflow checklist and the AI QA test case library. Keeping the dataset versioned and the rubric stable turns RAG evaluation from a one-time demonstration into an ongoing model evaluation practice.
