How to Audit AI Agents: A 5-Step Compliance Guide (2026)

You can't audit what you haven't catalogued. The first step is producing a complete, versioned inventory of every AI agent operating in your organization — including agents running in production, staging, and any third-party agents you've deployed that operate under your responsibility.

For each agent, document: the model(s) powering it, the version deployed, what external tools and APIs it can call, what data it has access to, who owns it internally, when it was last updated, and whether it's currently active. Include agents built on foundation models from external providers — if you're deploying it, you're responsible for it under the EU AI Act regardless of who built the underlying model.

Common failure mode: Teams discover 30–50% more agents than they thought existed. Shadow deployments — agents set up by individual teams without central oversight — are endemic. Use your infrastructure logs, not just your ticketing system, to find them.

Deliverable: AI Agent Registry spreadsheet with one row per agent, all fields above populated, sign-off from each agent owner confirming accuracy.

For every agent in your inventory, document exactly what decisions it can make autonomously versus what requires human approval. This is the most consequential step for high-risk AI compliance — the EU AI Act's human oversight requirements (Article 14) hinge entirely on whether effective oversight mechanisms exist for the decisions that matter most.

Create a decision authority matrix: list every action category the agent can take (send email, write to database, call external API, make financial transaction, modify access permissions, etc.) and mark each as: fully autonomous, autonomous with logging, requires notification, or requires approval. Then audit whether the actual implementation matches what's documented.

The gap you're looking for: actions that should require approval but are running autonomously. In practice, this is almost always present — agents get more capabilities over time without corresponding oversight upgrades.

Deliverable: Decision Authority Matrix per agent, reviewed and signed by legal and the relevant business owner. Any mismatches between documented and actual authority flagged for remediation.

Pull the last 90 days of logs for each agent and assess whether they meet the EU AI Act Article 12 standard: sufficient to reconstruct events and identify the causes of situations that gave rise to significant risks or incidents.

Standard application logs typically fail this test. They tell you an API call was made and whether it succeeded — not what the model was reasoning about, what context it had, what alternatives it considered, or what policy constraints were active. A compliant audit trail for an AI agent must capture: the full prompt context (or a sufficient summary), the tool calls made and their inputs/outputs, the agent's stated reasoning if available, timestamps with millisecond precision, model version and parameters, and the identity of any human who reviewed or approved the action.

Assess each agent's logs against five criteria: completeness (all actions captured), reconstructability (can you replay what happened?), tamper-evidence (can logs be altered without detection?), retention (are they kept for the required period?), and accessibility (can regulators access them within required timeframes?).

Deliverable: Audit Trail Assessment report per agent, scoring each on the five criteria above with specific gaps documented and remediation timelines assigned.

Documenting what your policies are and actually verifying they enforce correctly are two different things. This step runs active tests against each agent's policy boundaries — including adversarial scenarios — to confirm that restrictions work as documented.

Design a test battery covering three categories. Boundary tests: actions just inside and just outside the defined scope — confirm allowed actions succeed and prohibited actions are blocked. Rate limit tests: verify that frequency caps are enforced and that the 61st call in a 60-call limit is actually denied. Adversarial tests: prompt injection attempts, goal hijacking, and attempts to get the agent to misrepresent its capabilities or ignore its constraints. Document the test inputs, expected outcomes, actual outcomes, and any discrepancies.

Pay particular attention to edge cases that weren't anticipated when policies were written — these are the most likely failure modes. If an agent can call a financial API, test what happens if it tries to call it in a context where that action wasn't intended. If it can read email, test whether it can be prompted to exfiltrate email to an unauthorized destination.

Deliverable: Policy Enforcement Test Report with test cases, pass/fail results, and an actionable remediation list for any policy gaps found. This document is specifically requested by auditors under Article 15 adversarial testing requirements.

Steps 1–4 produce evidence. Step 5 assembles that evidence into the documentation package that regulators, auditors, and enterprise procurement teams will actually request. This is not a one-time deliverable — it's a living document that must be updated whenever a significant change is made to any agent in scope.

The core document set required by the EU AI Act Annex IV (technical documentation) for each high-risk agent includes: general description and intended purpose, system architecture and component dependencies, training methodology and data governance documentation, validation and testing results (including the outputs from Steps 3 and 4), performance metrics and known limitations, post-market monitoring plan, and residual risk disclosure. For enterprises deploying rather than developing agents, your compliance package also needs documented evidence of supplier due diligence — what you verified about the underlying models and platforms before deployment.

Structure the documentation so it can be produced quickly on request. Regulators under Article 64 have the right to request access to technical documentation, and organizations that can't produce it promptly face additional scrutiny regardless of whether their underlying compliance is solid.

Deliverable: Full Compliance Documentation Package per agent, versioned and stored with access controls. Include a documentation index that maps each document to the specific Article or NIST function it satisfies, so gaps are immediately visible.