The problem
The client had built a customer-facing AI agent with tool access into their internal systems (order lookup, refund processing, account modification). Four weeks before a planned public launch, their security team asked a question nobody on the product team could answer confidently: what happens if someone tries to manipulate this agent into doing something it shouldn't? They engaged us for an independent adversarial security review, because internal teams evaluating their own system tend to test the paths they already trust, not the ones an attacker would try.
This is a different discipline from typical AI development work. Building a reliable agent and finding the ways a motivated adversary can break it require different mindsets, and we were explicit that this engagement's job was to find problems, not to confirm the system was fine.
What we actually tested
We ran a structured adversarial review across several categories, not a single generic "try to jailbreak it" pass:
- Prompt injection through user-controlled content. Testing whether text a user could control (an order note field, a support message) could override the agent's system instructions or trick it into unintended actions. This is one of the most common real-world attack vectors for tool-using agents.
- Tool-use boundary testing. Systematically probing whether the agent could be manipulated into calling write-capable tools (refund processing, account changes) outside its intended authorization scope, including through indirect, multi-step social-engineering-style prompts rather than only obvious direct requests.
- Data exfiltration via the agent's own responses. Testing whether the agent could be induced to reveal information about other customers' accounts, internal system details, or its own system prompt and tool schemas. The last of these is itself a security-relevant leak.
- Excessive agency and confused-deputy scenarios. Cases where the agent, acting with the legitimate authority of the logged-in user, could be tricked into taking an action that technically falls within its permissions but clearly violates the user's actual intent, like processing a refund the user never asked for based on a manipulated conversation.
What we found
We found eleven distinct exploitable issues, three rated critical. The most serious: a crafted message in an order note field (content the agent retrieved and processed during normal order lookup) could inject instructions that got the agent to process an unauthorized refund. It was a genuine prompt injection vulnerability through a data channel the product team hadn't considered an attack surface, because it wasn't direct chat input.
A second critical finding involved the agent's error-handling path: a specific malformed request pattern caused it to fall back to a less-restricted tool-calling mode intended only for internal debugging, which had been left reachable in the production configuration. This is a common and easy-to-miss category of finding: not a flaw in the agent's core reasoning, but a configuration and deployment issue that only surfaces under adversarial testing, not normal QA.
Remediation and re-testing
We worked with the client's engineering team on fixes for each finding, prioritized by severity, then re-ran the adversarial test suite against the remediated system before sign-off. The order-note injection was fixed by treating all retrieved content, not just direct chat input, as untrusted and applying the same instruction-boundary enforcement to it. The debug-mode fallback was fixed by removing the code path entirely from the production build rather than just restricting access to it, because a restricted-but-present attack surface is still a surface.
Challenges and tradeoffs
- The order-note injection vector took real creativity to find, not a standard test. Most prompt injection testing focuses on direct chat input. Finding the indirect vector through a data field the agent processed during a normal tool call meant thinking through every piece of content the agent touched, not just what a user typed. That's a different testing mindset from functional QA.
- Balancing thoroughness against the client's tight pre-launch timeline was a real tension. A truly exhaustive adversarial review could run much longer than the four weeks available. We prioritized the highest-likelihood, highest-impact attack categories and were transparent that this was a risk-prioritized review, not an exhaustive one, so the client could make an informed launch decision.
- Some findings required product, not just engineering, tradeoffs. The debug-mode fallback existed because the engineering team had found it genuinely useful during development. Removing it required a product conversation about whether that convenience was worth the risk, not just a code fix. The client's leadership made that call once we'd laid out the tradeoff clearly.
Results
All eleven findings, including the three critical ones, were remediated and verified through re-testing before the client's public launch, which went ahead on schedule. The client's security team has since made the review a required gate for any new tool or data source added to the agent, adopting our recommendation that it be recurring.
What we'd do differently
We'd push to start this review earlier in the client's development timeline rather than four weeks before launch. Four weeks was enough to find and fix the critical issues, but starting earlier would have allowed a broader test surface and remediation without the pressure of a fixed launch date.