The problem
The client is a software company with around 4,000 employees and a twelve-person IT service desk. Most of the desk's volume was repetitive: access requests for internal applications, password and MFA resets, software installation requests, distribution list changes, and "how do I" questions. Median time to resolution for these was several hours, not because they were hard but because they waited in a queue behind one another. The desk wanted an agent that could actually complete this work, not a chatbot that links to knowledge base articles.
What we actually did
Tools first, prompts second
We started by defining eleven narrow tools the agent could call, each wrapping an existing admin API: look up a user, check group membership, request application access, trigger a password reset flow, reset MFA enrollment, queue a software install through the device management system, and so on. Each tool validates its inputs, enforces its own permission rules, and returns a structured result. The agent never receives broad admin credentials; it can only do what these tools allow.
A risk tier on every action
Every tool is assigned a risk tier. Low-risk actions, such as looking up information or installing pre-approved software, run immediately. Medium-risk actions, such as granting access to an application, route to the requester's manager for one-click approval in chat. High-risk actions, such as anything touching admin groups or finance systems, always create a ticket for a human engineer. The agent explains which path it took and why.
Evaluation on replayed tickets
Before launch we replayed 1,200 historical tickets through the agent in a sandbox with mocked tools and scored whether it picked the right tool sequence, asked for the right approvals, and correctly escalated what it should not handle. We tuned tool descriptions and the system prompt against this set until wrong-tool and missed-escalation rates were near zero, and we rerun it on every change.
Challenges and tradeoffs
- Prompt injection through ticket content. Users can paste anything into a request. Tool permissions are enforced server-side based on the verified requester identity, never on what the text claims, so an injected instruction cannot escalate privileges.
- Ambiguous requests. "I need access to the finance thing" maps to several systems. The agent asks one clarifying question with the likely options rather than guessing.
- Auditability. Every tool call, its inputs, the approval that authorized it, and its result are written to the existing ticketing system, so audits work exactly as they did before, just with more entries.
Results
Three months after launch, the agent resolves 64% of tier-1 requests end to end with no human touch, with a median time to resolution of about four minutes. The desk redirected the recovered hours toward a device refresh project that had been deferred for a year. There have been no privileged actions taken without the required approval, which was the security team's condition for launch.
What we'd do differently
We would instrument the "agent escalated to a human" path more heavily from day one. Those escalations are the best source of candidates for the next tool to build, and we initially logged them with too little detail to rank them.