AI Exception Management Audit: What Examiners Will Test

An AI exception management audit asks one question: when an automated decision process hit a case it could not or should not resolve alone, can the firm prove who decided what, on which evidence, and within which rules? A policy document does not answer that. An auditor expects a record for each exception, from detection through resolution, and expects it to have been created at the time, not reconstructed afterward.

This guide sets out what counts as an exception, how to classify and route it, which controls an auditor will test, what evidence to retain, and where US supervisory guidance ends and good practice begins.

TL;DR

• An AI exception is a case that needs judgment, lacks information, or calls for an authorized alternative route. Define the triggers before the agent runs, not after the first incident.
• Classify exceptions by the next action required (data correction, policy override, ambiguity resolution), and route each to a named, authorized decision owner.
• Retain contemporaneous evidence: inputs, model output, the human review, the resolution code and any change that followed.
• Auditors test whether controls are rigorous, effective and implemented. They do not redo model validation, and human review that merely approves can be treated as a box-ticking exercise.
• Much of the US expectation is supervisory guidance applied proportionately to firm size and risk, so document your reasoning for the scope you chose.

Table of Contents

• What counts as an exception in an AI-driven process
• Classifying and routing AI exceptions
• Controls from detection to resolution
• Evidence to retain for each exception
• How auditors test exception management
• Monitoring trends and updating controls
• Binding rules versus guidance in the US
• Perspective: why exception evidence belongs outside the agent
• Preparing evidence before the auditor asks
• FAQ
• Sources

What counts as an exception in an AI-driven process

An exception is a case that needs judgment, has missing information, or requires an authorized alternative route, according to Midpoint's guide to AI workflow exception management. The same source lists the usual triggers:

• missing or conflicting data;
• policy or scope boundaries;
• ambiguous intent;
• low confidence;
• safety-sensitive conditions.

A second practitioner source adds a useful framing: exceptions often arise where rules do not cleanly apply, where data is incomplete, or where the cost of being wrong is too high to automate (Flowrunner on automation exceptions).

For a regulated firm, the practical consequence is that an exception is not the same as a technical error. A timeout is an error. An agent that is asked to approve a fee waiver above its delegated limit, or that returns a credit recommendation with a confidence score below the firm's threshold, is an exception, because someone with authority must now decide. Auditors care about the second category because it is where accountability is exercised.

Write the trigger list as a control, not a description

An auditor reading a trigger list asks whether it is complete, whether it is enforced by the system, and who approved it. A list that exists only in a runbook fails the second question. Each trigger should have an owner, a version, a date of last review and a stated rationale, such as the policy limit or risk appetite statement it implements.

Classifying and routing AI exceptions

Technical error codes are a poor basis for routing. Effective programs categorize exceptions by the necessary next action, such as data correction, policy override or ambiguity resolution, and send each to the person or team authorized to resolve that specific ambiguity, with the evidence needed to decide (Midpoint).

A workable taxonomy for a financial services firm has four to six classes. For example:

The table is an illustration of the method, not a regulatory list. The point an auditor tests is that every class has a defined next action and a named owner.

Apply the taxonomy proportionately

The Conference of State Bank Supervisors' AI supervisory framework for examiners states that institutions should apply AI governance in proportion to their size, complexity, risk profile and use of AI. A community bank using one agent for document triage does not need the exception apparatus of a large institution running dozens of autonomous workflows. It does need to be able to explain why its scope is right for its profile.

Controls from detection to resolution

A defensible exception control has a lifecycle. Midpoint's guidance says controls should define stop rules, identify an authorized decision owner and escalation path, set acknowledgment and resolution targets, and record the resolution code and evidence (source). In practice, auditors walk through these steps in order.

1. Detection. The system flags the exception using a rule that predates the event. Test whether the rule fired consistently.
2. Stop rule. The agent halts or falls back to a safe path. Test that it cannot proceed while the exception is open.
3. Assignment. A named owner or queue receives the item with the context required to decide.
4. Acknowledgment and resolution targets. Time limits are defined, monitored and escalated when missed.
5. Decision. The owner approves, rejects, corrects or overrides.
6. Resolution record. A resolution code, the rationale and the supporting evidence are stored.

Make the human decision verifiable

The FINOS AI Governance Framework describes an approval gate in which the human decision is captured through an authenticated, replay-resistant mechanism that displays the resolved action, the triggering policy rule, the material consequences and independently sourced evidence. That standard is higher than a ticket comment saying "approved". It is also a good yardstick for what an auditor can verify without relying on anyone's recollection.

The same framework makes a point that matters for override decisions: a regulatory dimension alone does not turn a control function such as Compliance into an operational approver. Authority has to align with the institution's formal delegated-authorities framework. If your compliance team clears exceptions that the delegation matrix assigns to the business, you have created a segregation problem.

Beware review that proves nothing

The UK Financial Conduct Authority has warned that relying heavily on human review without addressing underlying model risk can mask model weaknesses, and that review can become a box-ticking exercise (FCA good and poor practice on AI in financial services). The FCA is a UK regulator, so this is not a US requirement, but US examiners ask the same question in different words. A reviewer who approves 99.8% of escalated items in seconds is evidence of a weak control, not a strong one.

Practical measures include sampling reviewer decisions for quality, tracking override and approval rates by reviewer, and measuring time spent per item. For design options on meaningful review, see these human-in-the-loop oversight models.

Evidence to retain for each exception

Control design in AI governance is moving upstream. EY's analysis of the 2026 COSO developments says firms must maintain contemporaneous audit evidence of the whole decision path, including inputs, model outputs, human interventions and exception resolutions, instead of relying on retrospective documentation (EY on COSO 2026 and AI governance).

For each exception, a complete record usually contains:

• the identifier of the agent, its version and its owner;
• the trigger rule that fired and its version;
• the inputs and model output, or references to them where privacy rules restrict copying;
• the assignment, acknowledgment and resolution timestamps;
• the identity and authority of the decision maker;
• the decision, the resolution code and the rationale;
• any follow-up change to data, routing, thresholds or the model.

The Federal Reserve's SR 26-2 notes that banking organizations commonly maintain comprehensive model information and that documentation can support tracking recommendations, responses and exceptions. Outside the US, the Monetary Authority of Singapore's guidelines on AI risk management go a step further for its institutions: document human oversight processes including incidents and near misses, and review interventions regularly to judge whether oversight works. Near misses are a habit worth adopting in any jurisdiction, because they show an auditor that the firm looks for weaknesses before they cause harm.

Evidence that survives scrutiny

Records are only useful if an auditor can trust them. Tamper-evidence, consistent timestamps and the ability to verify a record independently of the system that produced it all raise the weight of the evidence. Records that can only be read through the production application, or that an administrator can edit silently, invite sampling failures. Our related pieces on agentless audit trails and AI logging without customer data cover how to build this without copying sensitive content.

How auditors test exception management

Internal audit in US banking organizations is expected to evaluate independently whether AI risk management controls, including exception handling procedures, are rigorous, effective and correctly implemented, according to the OCC's Comptroller's Handbook on lending and loan portfolio risk management. SR 26-2 draws the boundary: internal audit generally should not duplicate model risk management activities such as development or validation. It evaluates whether practices are rigorous and effective and whether the related policies are implemented (SR 26-2).

That shapes the test plan. The auditor is not retesting your model. The auditor is testing the exception process around it.

Sampling the population

An auditor needs a complete population of exceptions to sample from. If exceptions are logged only when someone chooses to log them, the population is incomplete and every conclusion is weak. Test completeness by comparing the agent's raw activity metadata to the exception register. For method, see our approach to audit sampling for AI decisions.

Effective challenge

SR 11-7 describes effective challenge as critical analysis by objective, informed parties who can identify model limitations and produce appropriate changes (SR 11-7). Applied to exceptions, the test is whether someone independent of the team running the agent ever questions why exceptions occur and whether anything changes as a result. SR 26-2 also stresses clear roles, accountability and management of conflicts of interest, particularly between development and validation teams. Our note on segregation of duties for AI applies the same principle to agent operations.

Monitoring trends and updating controls

The exception queue is a data source. Midpoint recommends reviewing exception patterns to find recurring causes and then deciding whether to fix upstream data or integrations, improve evidence or routing, or add a narrowly tested automation rule (source). Each recurring pattern should lead to a documented decision, even if the decision is to accept the exception volume.

A monthly review pack might include:

1. exception volume by class and by agent;
2. median and 90th percentile time to resolution;
3. override rate and approval rate by reviewer;
4. top recurring causes and their remediation status;
5. exceptions that bypassed the intended route.

SR 26-2 adds that persistent deviations outside established performance thresholds may warrant model adjustment, recalibration or redevelopment (SR 26-2). A rising exception rate for one class is exactly that kind of signal, and the escalation from exception reporting to model owners should be written down.

Binding rules versus guidance in the US

Much of what auditors apply to AI exceptions comes from supervisory guidance and recognized frameworks, not from a statute written for AI. SR 11-7 and SR 26-2 are supervisory letters, and SR 26-2 says model risk management should be tailored to the banking organization's model risk profile and to the size and complexity of its operations. The CSBS framework likewise asks for proportionality. The COSO and FINOS materials are professional frameworks that firms adopt voluntarily.

This matters in two ways. First, you have room to design controls that fit your risk, but you must be able to justify the choices. Second, examiners and auditors treat guidance as the benchmark for safe and sound practice, so a deviation needs a documented reason. Confirm with your counsel or compliance function which instruments apply to your charter and regulator, since state and federal requirements differ by institution type.

Perspective: why exception evidence belongs outside the agent

We see the same gap across regulated firms. The agent platform records that an exception happened, but the evidence of who resolved it and under what authority lives in email, chat and ticket comments. When an auditor asks for a sample, the firm assembles the story by hand, and the story cannot be verified independently.

Our view is that exception evidence should be generated from metadata, outside the production path, and sealed so that it can be checked offline. That is why AETHER Pulse connects through OAuth metadata only, touches no customer data, and produces evidence packs signed with HMAC-SHA256. An inventory and identity graph of agents tells you which agents exist and who owns them, so an exception can be tied to an accountable owner. Risk concentration views, including financial blast-radius exposure, help decide which agents deserve the tightest stop rules.

We also hold that every classification should be deterministic and traceable to a public source. If a reviewer cannot see why an item was flagged, the control cannot be tested. Our articles on deterministic controls for AI and decision pipeline evidence explain the design in more detail.

— Eleye

Preparing evidence before the auditor asks

AETHER Pulse gives risk and compliance teams a read-only layer for inventorying AI agents and producing audit-ready evidence packs without inserting anything into production systems. Plans and the Article 26 readiness audit are described on our pricing page, and the Free, Pro and Enterprise plans are on our home page.

The Article 26 readiness audit is relevant if you also have EU operations, where deployer obligations for high-risk AI systems apply. For a US-only firm, the same evidence discipline supports examiner requests under the guidance discussed above. For a phased approach to rollout, see our 90 to 120 day agentless governance plan.

FAQ

What is the difference between an AI exception and an AI incident?

An exception is a case that the process was designed to hand to a human, such as low confidence or a policy boundary. An incident is an unplanned event that caused or risked harm, such as a malfunction. A well-run program logs both and links them when a repeated exception reveals a deeper fault.

Does the auditor need to review every exception?

No. Internal audit evaluates whether the exception process is rigorous, effective and implemented, and it typically does so by sampling from a complete population. Completeness of the exception register is the first thing to prove, because a biased population invalidates the sample.

Is human review of exceptions enough to satisfy regulators?

Not by itself. The FCA has cautioned that human review can mask model weaknesses and become a box-ticking exercise. Show that reviewers have authority, enough context and independent evidence, and that review results feed back into model and data fixes.

How long should exception records be kept?

Retention depends on your regulator, record-keeping rules and internal policy, so confirm the period with your compliance or legal function. Keep exception records at least as long as the decisions they relate to must remain reviewable. Store them in a form an auditor can verify independently.

Which US guidance applies to AI exception oversight?

Banking organizations look to the Federal Reserve's SR 11-7 and SR 26-2 on model risk management, to OCC handbook expectations on internal audit, and to frameworks such as COSO and the CSBS examiner guide. These are largely supervisory guidance applied in proportion to firm size and risk. Your regulator and charter determine the precise mix.

Sources

• AI Workflow Exception Management, Midpoint
• Automation exceptions, Flowrunner
• Artificial Intelligence Supervisory Framework, CSBS
• FINOS AI Governance Framework: human-in-the-loop approval gate
• AI in financial services: good and poor practice, FCA
• COSO 2026 and the shift in AI governance, EY
• SR 26-2, Federal Reserve
• SR 11-7, Federal Reserve
• Guidelines on Artificial Intelligence Risk Management, MAS
• Lending and Loan Portfolio Risk Management, OCC Comptroller's Handbook
• Human-in-the-loop oversight models, Dogtooth

Recommended

• AI Logging for Compliance Teams: Audit Evidence Without Customer Data
• Build Audit Ready Deterministic Controls for AI in Regulated Finance
• Map NIST to Evidence: Regulator Ready AI Reports for Compliance
• AI Governance Dashboard Requirements for Regulated Firms