LLM Safety Testing Checklist: Prompt Injection, Jailbreaks, and Data Leakage

Overview

An LLM application can fail even when its core prompt appears sensible. Risk may enter through user messages, retrieved documents, uploaded files, tool results, conversation history, model changes, or error-handling paths. A useful safety review therefore tests the complete application workflow rather than only the model’s direct answers.

The goal of LLM safety testing is not to prove that an application can never be misused. Instead, it is to identify realistic failure modes, measure their impact, add controls, and confirm that the controls continue to work after each meaningful change. Treat the process as part of model evaluation and production AI operations.

Before testing, document the application’s intended behavior. Record which users it serves, what data it can access, which tools it can call, what actions require approval, and what the system should do when it is uncertain. This baseline makes test results easier to interpret and prevents teams from treating every unusual response as an equally serious incident.

What to track

1. Prompt injection and instruction conflicts

Test whether untrusted content can override system or developer instructions. Place adversarial instructions in user messages, retrieved documents, web pages, email text, PDFs, tool responses, and other fields that the model may process. Examples include requests to ignore previous instructions, reveal hidden prompts, change the output format, or treat quoted content as a higher-priority command.

For each test, check whether the model distinguishes data from instructions and whether the application limits the effect of a successful injection. A robust design should not rely on the model alone: isolate untrusted content, validate tool arguments, restrict permissions, and require explicit authorization for consequential actions.

2. Jailbreak and unsafe-content attempts

Build a varied set of jailbreak tests rather than repeating one well-known phrase. Vary the language, tone, role-play framing, encoding, spelling, multi-turn context, and request format. Include direct requests for disallowed or harmful assistance relevant to your product’s risk profile, as well as attempts to obtain the same result indirectly.

Record whether the application refuses, safely redirects, asks for clarification, or produces actionable harmful content. Also test whether a refusal remains stable after repeated pressure and whether the user can bypass a safeguard by switching languages, using an attachment, or starting a new conversation.

3. Sensitive-data disclosure

Test for exposure of system prompts, credentials, API keys, personal information, confidential business data, training examples, retrieved documents, and other protected content. Use synthetic secrets and clearly marked test records where possible. Never place live credentials or unnecessary personal data in a red-team dataset.

Include direct extraction requests, indirect questions, membership-inference-style prompts, cross-user conversation tests, and requests that combine several harmless fields to reconstruct sensitive information. Verify both the model response and application logs, traces, analytics, caches, exports, and error messages.

4. Tool misuse and excessive permissions

Inventory every tool the model can call and define the narrowest permission required for each one. Then test malformed arguments, unauthorized targets, repeated calls, destructive actions, prompt-injected tool results, and attempts to use a low-risk tool as a path to a high-risk action.

Controls should exist outside the model prompt. Apply server-side authorization, schema validation, rate limits, destination checks, transaction limits, and human approval where appropriate. A model saying “I will not do that” is not a substitute for an API that rejects an unauthorized request.

5. Insecure fallback behavior

Force failures in the model API, retrieval layer, tool service, parser, timeout handler, and content filter. Confirm that the fallback does not expose raw errors, return unverified data, silently switch to a less restricted path, or claim that an action succeeded when it did not.

Test partial failures as well as total outages. The safest fallback may be a clear limitation message, a queued request, or a human review path. Make the behavior observable so operators can distinguish a safe refusal from an unavailable dependency.

Reusable test-case template

Store each case in a versioned test set with these fields:

• ID and category: for example, INJ-004 or TOOL-007.
• Threat scenario: what an attacker or careless user is attempting.
• Setup: model, prompt version, user role, retrieved context, tools, and permissions.
• Input: the exact message, document, attachment, or tool response.
• Expected behavior: refusal, redaction, approval request, safe answer, or blocked action.
• Observed behavior: response, tool calls, side effects, logs, and latency.
• Severity and status: impact, likelihood, owner, remediation, and retest date.

Cadence and checkpoints

Run a baseline safety review before the first production release. Include representative normal requests as well as adversarial cases; safety controls can reduce usefulness if they are too broad or trigger unpredictably. Keep the baseline dataset stable so results can be compared over time, and maintain a separate exploratory set for new attack ideas.

Use a lightweight automated regression suite on every change to prompts, model settings, routing, retrieval, tool schemas, permissions, filters, or fallback code. A smaller suite can check critical controls quickly, while a broader red-team run can happen before a major release.

Review the application monthly or quarterly according to its exposure and change rate. At each checkpoint, examine new incidents, blocked attempts, false refusals, tool-call logs, privacy findings, and changes in the data or user population. Repeat manual red teaming when automation finds a promising weakness or when the application gains a new capability.

Before a release, confirm that every high-severity finding is closed, accepted by an accountable owner, or blocked by a documented temporary control. Link safety cases to the relevant prompt, code, model configuration, and deployment so a failed test can be reproduced.

How to interpret changes

Use a severity-ranking matrix

Rank a finding by both potential impact and likelihood. A practical matrix can use four levels:

• Critical: a credible path to severe harm, broad data exposure, unauthorized high-impact action, or compromise of core controls.
• High: meaningful sensitive-data disclosure, repeatable unsafe assistance, or unauthorized tool use with limited containment.
• Medium: a narrower bypass, unreliable refusal, excessive information disclosure, or a failure that requires several unusual conditions.
• Low: minor policy inconsistency, harmless prompt leakage, or a confusing response with no meaningful security consequence.

Consider exploitability, affected users, data sensitivity, reversibility, detection quality, and whether the failure crosses a trust boundary. A low-frequency issue may still deserve urgent treatment if its impact is severe or the application can act without human review.

Separate model behavior from system behavior

When a test fails, identify where the control broke. The model may have generated an unsafe answer, but the application may also have passed untrusted content without separation, granted excessive permissions, skipped output validation, or logged confidential data. Remediation is stronger when it addresses the entire chain.

Track quality and safety together. A lower unsafe-output rate is not an improvement if the system now refuses legitimate work, hides errors, or routes more requests through an unmonitored fallback. Use the same representative cases after each fix and compare behavior across the models and configurations your production workflow supports.

Maintain a remediation log

For every finding, record the root cause, affected component, severity, owner, planned fix, temporary mitigation, validation test, and closure date. Reopen the issue if a model change, prompt revision, new tool, or data-source change causes regression. This log becomes the operational memory of the safety program and helps prevent repeated discoveries.

When to revisit

Revisit this checklist before every significant release and whenever you change the model provider, model version, system prompt, retrieval index, tool set, authorization rules, content filters, memory behavior, or fallback path. Repeat testing after adding new document sources, connecting external services, accepting file uploads, expanding user roles, or changing the kinds of actions the application can perform.

At a minimum, schedule a monthly or quarterly review that compares the current results with the previous checkpoint. Look for new attack patterns, changes in refusal consistency, increased sensitive-data exposure, unexpected tool calls, rising false refusals, and gaps in logging. Update the test set when an incident, support report, provider change, or red-team exercise reveals a new scenario.

For related release controls, see the production AI readiness checklist, the LLM evaluation framework, and the guide to prompt versioning and regression testing. Teams working with retrieval should also review how to evaluate a RAG system.

To put the process into practice, assign an owner, copy the test-case fields into your evaluation repository, select a small release-gate suite, and run it against the current production configuration. Fix or formally accept the highest-severity findings, preserve the results, and set the next review date before shipping.