Reliable structured LLM output requires more than asking a model to “return JSON.” This checklist explains how to design a JSON Schema, constrain generation where your provider supports it, validate syntax and types, recover from failures safely, and build regression tests for ambiguous inputs and future prompt changes.
Overview
Structured LLM output is useful when an AI response feeds another part of an application: a database insert, API request, workflow decision, search filter, or user interface. Plain text parsing is fragile because models may add explanations, omit fields, change a number into a string, or return a value that is syntactically valid but operationally unsafe.
A dependable workflow treats the model as one component in a validation pipeline:
- Define the contract. Specify the object shape, required fields, allowed values, and types.
- Constrain generation. Use a provider’s structured-output or JSON mode when available, while checking its schema limitations.
- Validate independently. Parse the response and validate it with a JSON Schema library in your application.
- Classify failures. Separate malformed JSON, schema violations, invalid business values, and transport errors.
- Recover deliberately. Retry only when the failure may be transient or correctable, and avoid repeated unbounded calls.
- Test representative cases. Keep fixed examples for normal, ambiguous, empty, adversarial, and unusually long inputs.
JSON mode can help produce syntactically valid JSON, but it does not automatically guarantee that every required field exists or that each value has the expected type. Application-side validation remains the final boundary before data is stored or acted upon.
For broader production planning, pair this workflow with the production AI readiness checklist and track quality, latency, cost, and reliability as separate concerns.
Checklist by scenario
Designing a new structured response
- Write down the smallest useful response shape. Avoid fields that the application does not use.
- Choose explicit JSON types:
string,number,integer,boolean,array, orobject. - Mark fields as required only when the application cannot proceed without them.
- Define whether an unknown value should be represented as
null, an empty array, or an omitted optional field. - Use enumerated values for controlled states such as
"approved","needs_review", and"rejected". - Document units, time formats, case sensitivity, and whether strings may contain Markdown or HTML.
A compact schema is easier to support across providers than one that depends on advanced or provider-specific features. For example:
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"additionalProperties": false,
"required": ["summary", "priority", "actions"],
"properties": {
"summary": {"type": "string", "minLength": 1},
"priority": {"type": "string", "enum": ["low", "medium", "high"]},
"actions": {
"type": "array",
"items": {"type": "string"}
}
}
}The additionalProperties setting is a design choice. Rejecting extra keys makes contract drift visible; allowing them can make gradual schema evolution easier. Choose deliberately and test the decision.
Calling a model that supports structured output
- Pass the schema through the provider’s supported structured-output interface rather than relying only on prompt instructions.
- Confirm which JSON Schema features the selected model and SDK actually support.
- Keep the prompt consistent with the schema. Explain what each field means, but do not create contradictory instructions.
- Request data only, without commentary, when the endpoint does not enforce a response schema.
- Record the model identifier, schema version, prompt version, and validation result with each request where operationally appropriate.
Provider APIs and SDK syntax change over time. Keep the integration behind a small adapter so that switching models or updating a client library does not spread structured-output logic throughout the application.
Validating and recovering from a response
Use a two-stage check. First, parse the raw response as JSON. Second, validate the parsed value against the schema and any business rules that JSON Schema does not express conveniently.
| Failure class | Example | Recommended response |
|---|---|---|
| Transport failure | Timeout, rate limit, or unavailable service | Apply bounded retry logic with backoff where appropriate; preserve the original request context. |
| Syntax failure | Truncated or fenced JSON | Retry with a clear correction instruction or use a repair path only if the repaired result is validated afterward. |
| Schema failure | Missing priority or wrong type | Send a targeted retry describing the validation error, with a strict attempt limit. |
| Business-rule failure | Date is valid JSON but outside the allowed range | Reject or route for review; do not assume a model retry will fix a missing source fact. |
A safe retry should be idempotent, bounded, and observable. If two attempts fail, return a controlled application error or a review state rather than silently passing unvalidated output downstream.
Testing a structured response
Keep a regression file with the input, expected constraints, and observed result. A useful template is:
{
"caseId": "missing-priority",
"input": "Summarize this request and identify its urgency.",
"mustBeValidJson": true,
"requiredFields": ["summary", "priority", "actions"],
"allowedPriority": ["low", "medium", "high"],
"expectedOutcome": "valid_or_controlled_failure"
}Include at least one ordinary case, a short input, a long input, missing context, conflicting instructions, unexpected language, empty input, and content containing braces or quoted text. Test both acceptance and rejection: a validator that accepts everything is not providing a useful contract.
What to double-check
- Required fields: Are fields required because the application needs them, or merely because they appeared in an early example?
- Null handling: Does the consumer distinguish between
null, an empty string, an empty array, and a missing key? - Type coercion: Will the validator reject
"42"when the application expects the number42? - Enums: Are values normalized, or can casing and spelling differences break downstream logic?
- Extra keys: Should unknown properties be rejected, logged, or ignored?
- Length limits: Could a model return an unexpectedly large string or array that affects storage and cost?
- Security boundaries: Is model-produced text displayed as HTML, executed as code, or inserted into a query? Validate and escape it for its destination.
- Partial output: Can streaming or a timeout produce an incomplete document? Treat incomplete output as invalid.
- Observability: Can you identify the schema version, prompt version, model, failure class, and retry count without logging sensitive input unnecessarily?
Schema validation is not a substitute for factual evaluation. A response can satisfy every type requirement and still contain an incorrect summary or an unsafe recommendation. For quality testing, connect structured-output checks with an LLM evaluation framework and maintain test cases as prompts and models change.
Common mistakes
- Relying on prompt wording alone. “Return valid JSON” is an instruction, not an enforcement mechanism. Use provider constraints and a local validator.
- Parsing before removing transport wrappers. Some SDKs return a message object, while others return text containing a code fence. Normalize the response at the integration boundary, then validate the resulting JSON.
- Blindly repairing malformed JSON. A repair library may produce parseable output that changes the intended meaning. Treat repair as a candidate result, never as proof of correctness.
- Retrying every failure. Retries cannot reliably solve missing information, contradictory requirements, or a poorly designed schema. Classify the error first.
- Changing the schema without versioning it. Adding a required field can invalidate old prompts, fixtures, and consumers. Use explicit schema versions and migration tests.
- Testing only happy paths. Ambiguous inputs and unusual content often reveal the real integration failures.
- Mixing extraction and interpretation without stating the difference. If a field is inferred rather than quoted from the source, name that behavior and evaluate it separately.
Keep prompt changes and schema changes reviewable. The prompt versioning and regression testing workflow provides a useful companion process for comparing changes before release.
When to revisit
Revisit this checklist before a production release, after changing a model or SDK, and whenever a provider changes its structured-output interface or schema support. It is also worth reviewing before seasonal planning cycles or other periods when input volume, document formats, or business rules change.
Return to the schema when a new field is added, an optional field becomes required, an enum gains a value, or a downstream consumer changes its expectations. Re-run the regression set after prompt edits, retrieval changes, temperature or sampling changes, and model substitutions. Compare not only validity rates but also retry frequency, latency, cost, semantic accuracy, and the number of responses routed for human review.
Use this final release checklist:
- Schema is versioned and stored with the application change.
- Provider constraints match the schema actually validated in code.
- Syntax, schema, and business-rule failures are distinguishable in logs and metrics.
- Retries are bounded, safe, and tested.
- Regression cases cover ambiguous, empty, long, and adversarial inputs.
- Unvalidated output cannot reach a database, external API, or user-facing workflow.
- Rollback behavior is defined for a failed schema or model change.
When the underlying model, prompt, schema, or workflow changes, repeat the same checks rather than assuming that a previously reliable response contract will remain reliable.
