A useful LLM evaluation dataset is not simply a large collection of prompts. It is a versioned, representative set of test cases that reflects real users, important failure modes, and the conditions in which your application must perform. This checklist explains how to design one, balance its sources, protect sensitive data, and keep it relevant as your product changes.
Overview
LLM evaluation dataset design starts with a clear evaluation question. You may be comparing models, testing a prompt change, validating a retrieval workflow, or checking whether a structured response meets an application contract. Each purpose requires slightly different test cases and metrics.
Define the unit of evaluation before collecting data. A unit might be a user message, a conversation with several turns, a question plus retrieved documents, or a complete request-and-response workflow. Record the expected behavior separately from the model's output. Without that separation, reviewers may unconsciously judge the answer against whatever the current system happened to produce.
A practical dataset should contain more than typical examples. Include routine requests, ambiguous inputs, difficult edge cases, known production failures, safety-sensitive prompts, and cases where the correct response is to ask for clarification or refuse. For a retrieval-augmented application, preserve the retrieved context so you can distinguish a retrieval failure from a generation failure.
Keep the dataset versioned. A simple version identifier, change log, owner, and review date make it possible to explain why a score changed. For a broader workflow, pair this dataset with a repeatable benchmark process such as the LLM evaluation playbook and run important cases through prompt regression testing.
A reusable dataset schema
{
"case_id": "support_0042",
"dataset_version": "2026-01",
"task": "answer_customer_question",
"input": "Can I change the delivery address?",
"context": ["retrieved document or conversation context"],
"expected_behavior": "Explain the permitted process or ask for order details",
"reference_answer": "Optional reference for comparison",
"risk_level": "medium",
"source_type": "historical",
"tags": ["policy", "clarification", "customer_support"],
"locale": "en",
"review_status": "approved"
}Not every field is required for every application. The important principle is to preserve enough metadata to filter results by scenario, risk, language, feature, and source. This turns a score into an explanation.
Checklist by scenario
When starting from scratch
- Write down the application's core tasks and out-of-scope requests.
- List the behaviors that matter: factuality, completeness, instruction following, tone, refusal quality, latency, or valid structured output.
- Create a coverage matrix with columns for task, user type, difficulty, risk, language, channel, and expected behavior.
- Seed the set with manually reviewed examples before adding synthetic cases.
- Reserve a portion of cases for final evaluation rather than repeatedly using every example during prompt development.
When using production examples or support tickets
- Sample across time, user segments, product areas, and outcome types instead of choosing only recent or memorable tickets.
- Include successful interactions as well as complaints and escalations; otherwise the dataset overrepresents failure.
- Remove or replace personal, confidential, account, and operational data before the examples enter evaluation systems.
- Preserve the original intent and relevant context while documenting any redaction or transformation.
- Have a qualified reviewer confirm the expected answer and label the failure mode.
When adding edge cases and adversarial inputs
- Test ambiguous wording, incomplete requests, contradictory instructions, and unusually long inputs.
- Include prompt injection, data leakage, jailbreak, and unauthorized-action scenarios where they match the application's threat model.
- Test attempts to override system constraints through quoted text, retrieved documents, tool results, or conversation history.
- Label whether the desired behavior is an answer, clarification, refusal, escalation, or safe alternative.
For a security-focused set, use the LLM safety testing checklist. For applications that return JSON or another contract, add schema-validity cases and consult the guidance on structured output evaluation.
Choosing test-case sources
| Source | Best use | Strength | Watch for |
|---|---|---|---|
| Manual curation | Defining critical behaviors and gold cases | High contextual judgment | Small volume and reviewer bias |
| Historical production data | Finding realistic language and failures | Strong connection to actual use | Privacy, sampling bias, and stale workflows |
| Synthetic generation | Expanding coverage and variations | Fast, controllable scale | Repetition and unrealistic assumptions |
| Crowdsourced cases | Testing wording and perspectives | Broader linguistic variety | Inconsistent labeling and domain knowledge |
Use sources together rather than treating one as sufficient. Manual cases can define the standard, historical cases can reveal what users actually do, synthetic cases can fill visible gaps, and crowdsourced cases can challenge internal assumptions.
What to double-check
Coverage and balance
Review the coverage matrix before each release. Ask which high-volume tasks are represented, which low-frequency but high-impact failures are included, and which user groups or languages are missing. A dataset can look diverse while still containing many near-duplicates of the same easy request.
Track counts by task, risk, difficulty, source type, and expected outcome. Do not let an abundant category dominate the overall score. Report important slices separately, particularly when an average could hide poor performance on a critical workflow.
Labels and reference answers
Define labels in operational terms. “Good answer” is too vague; “states the refund limit, avoids inventing an exception, and directs the user to the required verification step” is reviewable. Where several answers are acceptable, record criteria or key facts instead of forcing one exact sentence.
For subjective qualities such as tone or helpfulness, use a rubric with examples. Consider combining automated checks with human review, especially for consequential or ambiguous cases. The guide to LLM-as-a-Judge versus human review can help structure that decision.
Deduplication and leakage
Remove exact duplicates, near-duplicates, and paraphrases that would make a test set appear broader than it is. Compare normalized text, key entities, intent labels, and conversation context—not just character-for-character equality. Keep one representative case and record related variants when the variation itself is meaningful.
Check for leakage between development and final evaluation sets. If a prompt, reference answer, or case-specific instruction is repeatedly used during optimization, its score may no longer represent unseen performance. Maintain a protected holdout set with restricted access.
Privacy and traceability
Store the source and transformation history for every case. Use stable case IDs rather than embedding sensitive identifiers in filenames or tags. Redact personal data, secrets, internal URLs, and credentials, and verify that redaction did not remove the information needed to understand the task. Apply the access controls and retention practices appropriate to your environment.
Common mistakes
- Optimizing for volume: Thousands of weak or repetitive cases can be less useful than a smaller, well-labeled set with meaningful coverage.
- Testing only happy paths: Routine prompts do not reveal how the application handles ambiguity, missing context, refusal, or conflicting instructions.
- Using synthetic data as ground truth: Synthetic cases are useful for expansion, but review them for realism and label accuracy.
- Changing labels to fit outputs: Expected behavior should reflect the product requirement, not whichever response a current model produced.
- Reporting one aggregate score: Overall averages can conceal failures in high-risk or underrepresented slices.
- Ignoring the surrounding workflow: Evaluate retrieval context, tool calls, formatting, and error handling when those components affect the user-visible result.
- Leaving the dataset undocumented: Without ownership, version history, and inclusion rules, future reviewers cannot interpret score changes.
When to revisit
Revisit the dataset whenever the underlying inputs change. Schedule a review before major planning or release cycles, and trigger an unscheduled review after a severe production failure, a new feature, a change in target users, a model or provider migration, a new retrieval source, or a material prompt change.
At each review, compare recent failures with current coverage, retire cases that no longer represent the product, add new failure patterns, and rebalance overrepresented categories. Recheck privacy transformations and schema assumptions when evaluation tools or model APIs change. Keep retired cases in an archive when they document historical behavior, but do not let them silently influence the active benchmark.
Before the next release, use this short action list:
- Export or collect newly approved examples.
- Redact sensitive information and record transformations.
- Label task, risk, source, expected behavior, and relevant slices.
- Deduplicate against the active dataset.
- Review coverage and protect the holdout set.
- Run baseline and candidate systems with the same evaluation settings.
- Inspect regressions by slice, not only by aggregate score.
- Record the dataset version, results, reviewer decisions, and follow-up actions.
A representative LLM evaluation dataset is maintained, not finished. Treat it as part of the product's quality system, and it will remain useful as prompts, models, workflows, and user behavior evolve.