The problem
The client sells applicant tracking and recruiting software, with a growing base of customers in the European Union. Over several years they had added AI features: resume parsing, candidate ranking, job description drafting, interview scheduling, and more. Under the EU AI Act, AI systems intended for recruiting and selecting candidates, including filtering applications and evaluating candidates, fall into the high-risk category, which carries obligations for risk management, data governance, technical documentation, record-keeping, transparency, human oversight, and accuracy and robustness. Enterprise customers' procurement teams had started sending detailed AI questionnaires, and two large deals were stalled on them.
What we actually did
An inventory nobody had
We started by finding every AI feature in the product, including ones built by individual teams without central review. We found 17 and documented what each did, what data it used, which model powered it, and where its output went. Several were purely assistive (drafting a job description a recruiter edits); four directly influenced which candidates a recruiter saw or in what order.
Risk classification with counsel
Working with the client's legal counsel, who made the final legal determinations, we mapped each feature against the Act's categories. The four candidate filtering and ranking features were treated as high-risk. The assistive features carried lighter transparency obligations. That classification let the team focus effort where the regulation actually applied instead of treating every feature the same.
Controls built into the product
For the high-risk features we built four controls. Event logging records the model version, inputs, outputs, and the recruiter's action for every ranking. A human oversight design presents rankings as suggestions with visible explanations, and recruiters can review all applicants regardless of rank. Bias testing across available demographic proxies runs on held-out historical data before each model release. And a model change process requires evaluation results and sign-off before deployment.
Documentation that matches the system
We produced technical documentation for each high-risk feature covering intended purpose, data sources and governance, evaluation methodology and results, known limitations, and oversight measures. It's generated partly from the evaluation pipeline itself, so it stays current as models change instead of drifting from reality.
Challenges and tradeoffs
- Regulatory interpretation. Some provisions still depend on guidance and standards being finalized. We designed controls to the substance of the obligations and flagged open interpretation questions for counsel rather than guessing.
- Bias testing without demographic data. The client doesn't collect protected characteristics. We used carefully chosen proxies and synthetic test sets, and documented the limits of that approach explicitly.
- Customer and provider responsibilities. The Act places obligations on both the provider and the organizations that deploy the system. We produced customer-facing instructions for use so deployers know their own oversight responsibilities.
Results
Nine weeks after kickoff, the client had a complete AI inventory, a counsel-reviewed risk classification, the logging and oversight controls live in production, and technical documentation for each high-risk feature. They answered the stalled procurement questionnaires with evidence rather than promises, and both enterprise deals closed the following quarter. New AI features now go through the same classification and release process before they ship.
What we'd do differently
We would run the AI feature inventory as a standing process from the start. Discovering four ranking-related features built independently by different teams was the most time-consuming part of the project, and a lightweight registry would have prevented it.