AI · New service
FZ AI Auditor: audit of deployed AI agents
Scope, value realized and blast radius: three questions most organizations cannot answer with evidence.
The problem
AI's advantage is that it scales a solution to every case at once. That is also its exposure: it scales whatever the solution actually is, mistakes included, and some mistakes are critical. Almost every organization can list the agents and automations it has deployed. Almost none can say, with evidence, what those agents actually do versus what they were authorized to do, whether the result is worth what it costs, and how much damage one bad decision could do before a human notices. FZ AI Auditor answers those three questions with evidence, on a cadence.
Scope fidelity
What does the agent do, compared with what it was allowed to do?
Source: execution traces and logs, not the design document.
Value realized
What does it cost to run, and what does it really save?
Running cost, cost avoided and quality against a human baseline.
Blast radius
If it is wrong, how far does it go, and for how long?
Number of cases, severity of each, time before anyone notices.
Containment fit
Do the controls in place match what the risk demands?
Each control is verified, not assumed.
What the audit covers
Scope fidelity
What the agent was authorized to do, versus what it is actually doing in production, drawn from execution traces and logs rather than the design document. Scope drift is the normal state of a deployed agent; the real question is whether anyone is watching it.
Value realized
Volume handled, cost to run (compute, oversight, incident cleanup) against cost displaced, and a sampled quality review against a human-only baseline. A real number, not the business case that justified the build.
Blast radius
If this agent is wrong, how many cases does it touch, how severe is each one (reversible or irreversible, routine or financial, safety or compliance-critical), and how long before a human would catch it? This is the RTO/RPO question applied to an agent: how much can go wrong, and how fast can it be stopped.
Containment fit
The controls in place (approval gates, canary rollout, sampling review, a kill switch, a human owner) checked against what the blast-radius rating actually requires, not against what was built for convenience.
Why it matters
Mistakes multiply, not just output
A human error is one case. An agent error is every case it touched before someone looked: the same defect, repeated at whatever volume the agent runs at.
Value claims age fast
The business case that justified an agent six months ago is rarely re-tested. Scope drift and model or prompt changes move the real number without anyone re-running it.
Regulators are starting to ask the audit question
The EU AI Act's post-market monitoring duties (like the other high-risk obligations, now due from 2 December 2027 for Annex III and 2 August 2028 for Annex I, after the Digital Omnibus), ISO/IEC 42001, the NIST AI RMF and model-risk regimes already in force for financial institutions all expect an organization to show what a deployed model or agent does and how it is overseen, not just that it was validated once before launch.
Our engagement
Durations are indicative; they depend on the number of agents in scope and on the availability of evidence.
- 1-2 wksInventoryAuthorized vs observed
- 2-3 wksValueReal cost vs value
- 2-3 wksBlast radiusSeverity and detection
- 1 wkScorecardRanked by exposure
- QuarterlyCadenceOn every change
- 01
Inventory and scope map
1 to 2 weeks
Catalog every deployed agent and automation in scope. Compare authorized scope against observed behavior from logs, traces and, where present, the Citadel or gateway control plane.
- 02
Value ledger
2 to 3 weeks
Build the real cost-versus-value number per agent, with a sampled accuracy review against a human baseline.
- 03
Blast-radius assessment
2 to 3 weeks
Rate each agent's failure severity and detection time. Pressure-test containment controls against that rating, including a failure-injection exercise where the risk rating warrants it.
- 04
Scorecard and roadmap
1 week
A per-agent audit scorecard (scope fidelity, value realized, blast radius) and a remediation list ranked by exposure, not by ease.
- 05
Ongoing audit cadence
Quarterly or on material change
Re-run the scope map and blast-radius rating whenever a model, prompt or permission changes, and track drift between cycles instead of re-discovering it from scratch.
What you get
- Agent inventory with authorized and observed scope for each one
- A value ledger per agent: cost to run against value displaced, with the accuracy sample behind it
- A blast-radius rating per agent, with time to detection and worst-case severity
- A containment-gap list, mapped to what the rating actually requires
- An audit scorecard and a prioritized remediation roadmap
- A repeatable scorecard template for your own team's quarterly re-run
Standards we align to
Why FZ Consultation
We apply the same discipline (assess, test, measure, rather than take the design on faith) to business continuity, disaster recovery and cybersecurity for financial services, healthcare, energy and the public sector, since 2012. AI agents are becoming critical systems: they have access, they make decisions, and they fail. We audit them the way we audit anything else that can take the business down, and we hand the scorecard and the method to your own team rather than building a dependency on us.
Frequently asked questions
Why audit an agent that was validated before launch?
Because validation describes the agent on launch day. In production, scope drifts, models and prompts change and volumes move. The audit measures the agent as it is today.
What if we have almost no traces or logging?
The audit rests on evidence. If it does not exist, we discuss it at scoping: the absence of evidence is itself an important finding, and we agree with you on the best way to close that gap before or during the audit.
Do you test on our production systems?
It depends on your risk tolerance. At scoping we choose with you the approach for the failure-injection exercise: a staging or shadow copy, or a simulation from logs.
How much does it cost?
It depends on the number of agents and their risk level. We discuss it in a free 30-minute intro call and follow with a written proposal.
How is this different from AI governance?
Governance builds the framework: policies, inventory, roles, platform. The audit checks, with evidence, that the agents actually deployed respect that framework and are worth what they cost. The two complement each other and we can run them together.
What does our team keep at the end?
The scorecard and the method. We hand over a repeatable template so your team can re-run the audit each quarter, rather than building a dependency on us.