What should you test before launching an AI agent with tool access?

Ten categories matter most: mapping every data source the agent touches, direct and indirect instruction override, tool-use boundary testing, data exfiltration risks, excessive agency scenarios, error and fallback path auditing, logging sufficiency, human review requirements for high-stakes actions, and re-testing after remediation.

Is AI agent security testing different from normal QA?

Yes. Functional QA tests whether the system works as intended. Adversarial security testing specifically tries to make it do what it is not supposed to do, including through indirect vectors like data fields the agent processes, not just direct chat input. That takes a different mindset and skill set than standard testing.

AI Agent Security Checklist: What to Test Before You Launch

Most AI agent launches get functional QA (does it do what it's supposed to do) and skip adversarial testing (can it be made to do what it's not supposed to do) entirely. These are different disciplines, and the second one is where the real production incidents come from. Here's the checklist we actually run before we sign off on a launch.

1. Map every data source the agent touches, not just the chat input

List every place content enters the agent's context: direct user messages, retrieved documents, database fields, API responses, uploaded files, anything. Indirect prompt injection through a data source nobody flagged as an attack surface is consistently the most commonly missed vulnerability category. If it's not on this list, it hasn't been tested.

2. Test direct instruction override

Try to get the agent to ignore its system instructions through direct conversation: "ignore previous instructions," "you are now in developer mode," and variations. This is the most basic test and the one most teams already run, but it's worth confirming explicitly rather than assuming.

3. Test indirect injection through every data source from step 1

For each data source that feeds the agent, test whether embedding an instruction in that content can manipulate its behavior: a note field, a document title, metadata, anything the agent reads as part of a normal tool call. This is where the highest-severity findings tend to live.

4. Test tool-use boundaries

For every tool the agent has access to, test whether it can be manipulated into calling that tool outside its intended scope or authorization, including through indirect, multi-step social-engineering-style prompts, not just direct requests. Can a support agent be tricked into processing a refund it shouldn't? Can a scheduling agent be tricked into accessing a calendar it shouldn't see?

5. Test for data exfiltration

Check whether the agent can be induced to reveal information it shouldn't: another user's data, internal system details, or its own system prompt and tool schemas. System prompt exposure specifically can hand an attacker a roadmap for further attacks.

6. Test excessive agency scenarios

Look for cases where the agent, acting within its technical permissions, could be manipulated into taking an action that's authorized but clearly not what the user intended (a confused-deputy scenario). This category requires thinking through intent, not just permission boundaries.

7. Audit fallback and error-handling paths, not just the happy path

Check what happens when the agent hits an unexpected input, a malformed request, or an error condition. Debug or less-restricted modes left reachable in production configuration are a real, easy-to-miss finding category, and they don't show up in normal functional testing because normal testing doesn't try to trigger error states deliberately.

8. Confirm logging captures enough to investigate an incident after the fact

If something goes wrong, can you trace exactly what instruction or data led to the agent's action? Logging that only captures the final output, not the reasoning path and source content, makes post-incident investigation much harder.

9. Confirm human review is in place for genuinely high-stakes actions

For actions with real cost or safety consequences, decide explicitly whether the agent should ever be allowed to act autonomously, or whether human confirmation should be required regardless of confidence. This should be a deliberate decision, not a default.

10. Re-test after remediation, and plan to test again when capabilities change

A fix that hasn't been re-tested is unverified. The injection surface changes every time you add a new tool or data source, so a pre-launch review is a snapshot, not a permanent guarantee. Build recurring review into your process, tied to when the agent's capabilities change, not a fixed calendar schedule.

What this catches in practice

A well-run adversarial review against a customer-facing agent typically surfaces a real, non-trivial number of exploitable findings, including issues like an indirect injection vector hiding in an unexpected field (an order note, a support ticket body) or a debug-mode fallback path left reachable in production. This is exactly why the review needs to happen before launch, not after. Internal QA focuses on whether the agent works, so it consistently misses these categories; finding them requires deliberately trying to break the agent rather than confirming that it functions.

How we run this

We treat this as an independent review, not a self-assessment, since internal teams evaluating their own system tend to test the paths they already trust. A structured review across all ten categories, with a prioritized findings report and remediation support, typically takes three to five weeks depending on system complexity.

AI agent security checklist: what to test before you launch