What Happens When Your AI System Goes Wrong: A Governance Playbook for AI Incidents

Why Most Teams Are Completely Unprepared for AI Failures

AI incidents aren’t rare. They’re accelerating.

According to Stanford University’s AI Index Report 2025, documented AI safety incidents surged from 149 in 2023 to 233 in 2024 — a 56% increase in a single year. That’s not a slow creep. That’s a category of operational risk that’s growing faster than most organizations’ ability to manage it.

And yet, a World Economic Forum report published in 2025 found that escalation processes for identifying and reporting AI incidents across enterprises remain inconsistent and reactive. Most companies still have no standardized incident response protocols for AI failures. When something goes wrong, they improvise. Improvisation under pressure is a governance failure waiting to be compounded.

The challenge is also more nuanced than traditional software failures. When a server crashes, the failure is obvious and binary. When an AI system fails, the failure is often gradual, ambiguous, and contested. Did the model give wrong information, or did the user misinterpret correct information? Was that denied insurance claim a system error or legitimate risk scoring? Was the chatbot’s advice legally binding or just a suggestion? These questions don’t have obvious answers — which is exactly why you need a governance playbook that defines answers in advance, not in the middle of a crisis.

This article is that playbook.

What Actually Counts as an AI Incident

Before building a response framework, it helps to define what you’re responding to. “AI incident” gets used loosely, but there are meaningfully different failure categories — each with different response paths.

Operational and Safety Failures

These happen when the AI system behaves in ways misaligned with what its designers intended or what users reasonably expected. An AI coding assistant that deletes a database when asked to “clean up old files.” A chatbot that gives a passenger wrong refund information and creates legal liability. A recommendation engine that stops surfacing relevant results and causes measurable revenue drop.

The Future Society’s 2025 analysis of AI incident categories documented examples including an AI agent asked to check egg prices that instead purchased eggs without user consent (February 2025), a customer support bot that fabricated an explanation in response to a user complaint (April 2025), and an AI coding assistant that reorganized files in a way that made them inaccessible to both the human and the AI until outside help arrived (July 2025). These aren’t exotic edge cases. They’re the kind of failures that happen when autonomous AI systems are given insufficiently bounded task definitions.

Output Accuracy and Hallucination Failures

These occur when the AI produces factually incorrect, fabricated, or misleading outputs that users act on. The Air Canada chatbot case (Moffatt v. Air Canada, 2024 BCCRT 149) is the most legally significant example to date in consumer-facing AI. Air Canada’s chatbot told a grieving passenger he could purchase full-price tickets and retroactively apply for a bereavement discount. He followed that advice. The discount was denied. He took Air Canada to the British Columbia Civil Resolution Tribunal — and won, recovering $650.88 plus fees.

The legal principle established was significant: the tribunal ruled that Air Canada could not disclaim responsibility for its chatbot’s statements. Air Canada had actually argued the chatbot was “a separate legal entity responsible for its own actions” — a line the tribunal called “a remarkable submission.” The court was clear: the chatbot is part of Air Canada’s website, and the company is responsible for everything on its website.

That ruling established a precedent relevant to every company operating an AI-powered customer-facing system: you own your AI’s outputs, legally and operationally.

Performance Degradation Failures (Drift)

These are the slow ones — harder to detect but often larger in aggregate impact. A model that was accurate at deployment gradually loses accuracy over months because the real-world data it’s seeing has shifted. No alarm fires. No error log populates. Users don’t complain — they just get subtly worse outcomes. This category was covered in depth in the previous article on behavioral drift. From a governance standpoint, performance degradation counts as an incident once it crosses a threshold you’ve defined in advance — which brings up the first governance gap most organizations have.

Security and Adversarial Failures

Prompt injection attacks, jailbreaks, data exfiltration through LLM APIs, and manipulation of model behavior through adversarial inputs. According to Adversa AI’s 2025 analysis, generative AI was involved in 70% of AI security incidents reviewed, while agentic AI caused the most severe failures including crypto thefts, API abuses, and legal disasters. IBM’s data shows that one in five organizations experienced a breach related to unauthorized or shadow AI use.

Reputational and Compliance Failures

These don’t always involve a technical malfunction. Sometimes the AI worked as designed but the design itself was the problem. The nH Predict algorithm used by a major US health insurer to deny Medicare coverage was reportedly working as built — the system had a 90% error rate on appeals, meaning 9 out of 10 times a human reviewed the AI’s denial decision, they overturned it. The algorithm wasn’t broken. The design was the incident.

The Incident Response Gap: Why Companies Keep Getting Caught Flat-Footed

Most organizations have incident response protocols for cybersecurity events. Very few have separate protocols for AI-specific failures — and the difference matters.

A cybersecurity incident typically has a clear adversarial cause, clear technical evidence, and clear containment steps. An AI incident often has ambiguous cause (was it the model, the data, the prompt, the user context?), probabilistic evidence (it got worse, not wrong in a binary way), and contested containment options (do you rollback, retrain, throttle, or disable?).

The WEF’s 2025 Responsible AI Innovation report identified a structural problem that’s worth naming directly: limited incentives to report on AI incidents. Companies fear reputational and liability exposure for disclosure, so incidents get quietly managed rather than properly documented. This creates two compounding problems — the organization doesn’t learn from the failure, and no industry-wide signal exists to help others avoid the same issue.

One data point that illustrates how unprepared most organizations are: IBM reports that only 24% of current generative AI projects in production include security or risk measures at the time of deployment. Three-quarters of deployed AI systems go live without even basic safety controls, let alone incident response playbooks.

That needs to change. Here’s how.

Building the AI Incident Playbook: The Full Framework

Step 1 — Define Your Incident Tiers Before Anything Breaks

The most important governance work happens before an incident occurs. Specifically: you need to define what counts as an incident at your organization, and what tier of response each type triggers.

A practical three-tier model:

Tier 1 — Monitor and Log Threshold-crossing events that require documentation but not immediate escalation. Examples: output accuracy drops 5% below baseline, user complaint volume increases 20% above normal, input feature distribution shifts moderately (PSI between 0.1 and 0.25). Action: automated log entry, weekly review by model owner.

Tier 2 — Investigate and Contain Events requiring active investigation and possible operational changes within 24 hours. Examples: specific documented case of harmful output reaching a user, accuracy drops 15%+ below baseline, feature distribution shifts significantly (PSI above 0.25), any output that creates potential legal liability. Action: model owner escalates to product and legal within 4 hours, investigation opens, feature may be scoped or throttled.

Tier 3 — Escalate and Communicate Events requiring executive involvement, external communication, and potentially full system shutdown. Examples: confirmed harmful outputs affecting multiple users, regulatory compliance breach, media inquiry received about AI failure, safety-critical outputs in healthcare or financial systems. Action: CAIO or equivalent takes ownership within 1 hour, legal and communications engaged immediately, rollback or shutdown decision made within the same business day.

Step 2 — Assign Ownership Before the Incident

One of the most consistent findings across post-mortems of AI failures is that nobody owned the incident decisively when it happened. Data scientists assumed product owned it. Product assumed engineering owned it. Engineering assumed data science owned it. By the time ownership was resolved, the incident had escalated.

Every deployed AI system should have a named Model Owner — the person who is accountable for its behavior in production. This is separate from the person who built it. The Model Owner is responsible for monitoring, for receiving alerts, and for being the first call when something goes wrong.

Beyond the Model Owner, define a response team structure in advance:

The AI Incident Commander takes over for Tier 2 and Tier 3 events. This is typically a senior product or engineering leader with authority to make containment decisions (throttle, rollback, disable) without waiting for a committee.

The Legal Lead is engaged for any Tier 2 event that involves user-facing outputs and any Tier 3 event automatically. The Air Canada case is the clearest reason why: the legal question of what your AI outputs represent contractually or tortiously is not something to work out in real time during an incident.

The Communications Lead is pre-designated for any Tier 3 event requiring public-facing statement. The default should be one spokesperson, one voice. Communication vacuum in an AI incident gets filled by speculation and press reports — which are almost always worse than a brief, accurate disclosure.

Step 3 — Document the Incident Formally

Poor incident documentation is one of the most persistent governance failures in AI, partly because the immediate pressure is to fix the problem, not document it.

An AI incident record should capture, at minimum:

The timestamp of detection and the timestamp of actual incident onset (often different). The system and model version affected. The nature of the failure — what outputs were produced that shouldn’t have been, or what performance threshold was crossed and when. The user or data population affected and the estimated scale. The immediate containment action taken and by whom. The root cause hypothesis and the evidence supporting it. The post-incident action taken (retraining, rollback, scope limitation, user notification). The outcome of that action.

That documentation becomes your audit trail if a regulator asks, your learning record for the next incident, and your evidence in any legal proceeding. Under the EU AI Act, organizations operating high-risk AI systems are required to maintain this kind of record. Even outside regulated contexts, treating documentation as non-negotiable is the difference between organizations that improve and organizations that repeat the same failure.

Step 4 — Contain First, Investigate Second

This sequence matters and gets reversed more often than it should.

When an AI incident is confirmed at Tier 2 or above, the first question is: is this system still causing harm right now? If yes, contain it. Then investigate why.

The containment options available depend on what the system is:

For an LLM-powered customer-facing feature: scope limitation (restrict the topics or query types the model handles), add guardrails (filter outputs before they reach users), rate limiting (reduce volume while investigation proceeds), or full feature disable.

For a predictive ML model: continue running predictions but hold outputs in a review queue before actioning them, revert to a previous model version if one exists and performed adequately, or fall back to a rules-based system temporarily.

For an agentic AI system with real-world actions (purchasing, communication, data modification): immediate suspension of autonomous action capability. Agentic systems that take irreversible actions in the real world have zero tolerance for continued operation during an active incident. The egg-purchasing agent mentioned earlier is a mild example. An agentic system that could send contractual communications, process financial transactions, or modify enterprise data has a dramatically higher stakes failure mode.

One governance principle worth stating explicitly: an alert without a shutdown path is incomplete governance. Every AI system deployed in a consequential context should have a tested kill switch — a known, rehearsed procedure to disable it. If that procedure has never been tested before an incident, it will fail when most needed.

Step 5 — Root Cause Analysis (The Part Most Teams Skip)

Once the immediate incident is contained, the temptation is to declare it resolved and move on. This is the governance failure that leads to repeat incidents.

A structured root cause analysis for an AI failure should work through four layers:

Layer 1 — What happened technically? Did input data distribution shift? Did the model’s prediction confidence collapse in a specific feature subspace? Did a third-party API update change model behavior? Did a prompt structure change in production that the model handled differently?

Layer 2 — What process failed to catch it? Was there no monitoring for this type of failure? Were there alerts but no owner watching them? Was there a deployment without adequate pre-production testing? Did no one define the failure threshold in advance?

Layer 3 — What organizational factor enabled the gap? Was model ownership unclear? Did the team lack ML observability tooling? Was there insufficient time or budget allocated to production monitoring? Did the team treat deployment as a finish line rather than a starting point?

Layer 4 — What’s the systemic fix? Patching the specific failure is insufficient. What process change, tooling addition, or ownership clarification prevents the entire category of failure from recurring?

The NIST AI Risk Management Framework — released in January 2023 and updated with a Generative AI Profile in July 2024 — organizes AI risk management into four functions: Govern, Map, Measure, and Manage. Root cause analysis sits squarely in the Manage function, which covers responding to, recovering from, and communicating about AI incidents. But NIST’s framework correctly notes that Manage only works if the upstream functions — particularly Govern (ownership structures, policies, risk appetite) and Measure (monitoring, baselines, detection) — are already in place.

That’s the honest observation from working with production AI systems: most organizations that struggle with AI incidents struggle because they skipped Govern and Measure, not because they lack Manage protocols.

The Legal Layer: What AI Incidents Actually Cost You

The financial and legal consequences of AI incidents are real, documented, and in several cases, precedent-setting.

The Air Canada case established that companies are liable for AI chatbot outputs. The tribunal’s reasoning was clear: the chatbot is part of the company’s website, and the company is responsible for information on its website. “Liability is not avoided by automating the actions in question.”

The case also revealed a failure mode specific to AI deployment: Air Canada’s chatbot gave advice that contradicted the company’s own official policy page. The chatbot wasn’t drawing from the right source. That’s a governance failure at the integration and testing stage — not a model quality failure. A proper pre-deployment review would have tested whether the chatbot’s outputs aligned with official policy documentation.

The financial numbers on the broader landscape are significant. Stanford’s AI Index documented 233 harmful AI incidents in 2024 — up 56% from the prior year. Aon’s 2026 AI Risk analysis notes that more than 90% of insurance decision makers now consider AI-driven incidents a material concern. IBM’s data shows that shadow AI-related breaches cost organizations an average of $670,000 more than regular data breaches.

And the deepfake fraud case involving engineering firm Arup shows what the upper bound looks like: a finance employee was deceived into transferring $25 million via a deepfake video call impersonating the CFO and several senior colleagues. The entire call, except the victim, was AI-generated.

These are not statistics about hypothetical future risks. They’re documented outcomes from the last 24 months.

Industry-Specific Governance Considerations

Healthcare and Life Sciences

AI outputs in clinical contexts carry the highest stakes. The health insurance algorithm that systematically denied Medicare coverage — with a 90% overturn rate on appeals — represents what happens when a model’s outputs are actioned at scale without adequate human oversight. Under the EU AI Act, AI systems used to evaluate health conditions or inform healthcare decisions are classified as high-risk, requiring transparency documentation, logging of all decisions, and human oversight mechanisms before any automated decision can be acted on.

The governance principle: for any AI system where outputs directly affect patient care decisions, a human review gate is non-negotiable. The efficiency gain from automated decision-making is not worth the liability and patient safety cost of removing human accountability from consequential medical choices.

Finance and Lending

Automated credit scoring and fraud detection models carry regulatory obligations under existing frameworks (SR 11-7 model risk management guidance for US banks, for example) as well as emerging AI-specific regulation. The governance requirement of independent model validation — having someone other than the model’s builders evaluate its performance and risk profile — is not a new concept in banking, but it’s one that many organizations outside traditional finance haven’t yet adopted for their AI systems.

When a fraud model misses new attack patterns (concept drift) or a credit model becomes less accurate as economic conditions change (data drift), the regulatory expectation is that the organization had monitoring in place, detected the degradation, and took remediation action. “We didn’t know” is not an adequate response.

Customer-Facing Chatbots and Virtual Assistants

The legal lesson from Moffatt v. Air Canada applies to every company operating a customer-facing AI system: you are responsible for what it tells users. That means outputs need to be tested against authoritative internal documentation, monitored in production, and subject to escalation when users report harm.

NYC’s “MyCity” chatbot, launched to help small business owners navigate city regulations, was documented giving advice that contradicted city law — telling shop owners they could go cashless despite a 2020 law requiring acceptance of cash, and incorrectly advising on rental assistance discrimination. These failures weren’t model quality issues — they were governance failures. The system wasn’t properly tested against the actual regulatory content it was supposed to represent.

Agentic AI (The Frontier Risk)

Agentic systems that take autonomous real-world actions represent the highest-risk category for AI incidents, partly because their failures are often irreversible. IBM’s 2025 Agentic AI Governance Playbook notes that governance of agentic systems requires shifting focus from validation to control — from validating whether an answer is correct to controlling what actions an agent can take.

The critical governance controls for agentic systems: strict action scope definition (what categories of action can the agent take, and what requires human approval), reversibility requirements (preference for reversible actions over irreversible ones), human-in-the-loop gates for high-stakes or irreversible actions, and tested shutdown procedures.

Communication Protocols During an AI Incident

Most governance playbooks spend 90% of their space on technical response and 10% on communication. That ratio should probably be closer to 60/40, because how an organization communicates about an AI incident often determines the reputational and legal outcome as much as the technical response does.

Three principles for AI incident communication:

Designate one spokesperson, pre-incident. During a Tier 3 event, the number of people authorized to make public statements about the incident should be exactly one. Everyone else’s job is to support that person with accurate information and legal guidance. Organizations that allow multiple voices during an AI incident create conflicting narratives that get amplified.

Communicate facts, not interpretations. The initial statement during a live incident should be limited to what is factually confirmed: what system was affected, what type of failure occurred, what immediate action has been taken. Interpretations of cause, scope, or impact that turn out to be wrong in follow-up reporting are far more damaging than initial statements that acknowledge the limits of current knowledge.

Don’t disclaim your way out of accountability. The Air Canada “separate legal entity” argument is the clearest example of what not to do. Publicly attempting to disclaim responsibility for your AI’s outputs before a legal determination is made damages trust, signals poor governance, and often doesn’t even work legally. Acknowledge the incident, state the action taken, and let the investigation determine cause and accountability.

Here you can check decision ownership and liability frameworks for AI outcomes. Check our latest shadow AI warning signs for risk detection.

The Regulatory Context: What the EU AI Act Actually Requires

The EU AI Act is the most structured AI governance regulation currently in effect, phasing in obligations from 2025 through 2027.

For high-risk AI systems — which include AI used in critical infrastructure, medical devices, credit scoring, employment decisions, and others — the Act requires post-market monitoring, meaning active logging and review of system behavior after deployment. It requires logging of AI outputs sufficient to enable post-hoc assessment of compliance. It requires human oversight mechanisms for decisions that significantly affect individuals. And it requires incident reporting to relevant authorities when a high-risk AI system causes harm or performs in unexpected ways.

These aren’t aspirational guidelines. They’re legal obligations with enforcement mechanisms. Organizations deploying high-risk AI in EU markets that lack monitoring infrastructure, incident documentation practices, and response procedures are operating out of compliance.

For organizations outside the EU, the regulatory direction is clear: the EU AI Act sets the global standard that others are following. California’s AI regulations, Colorado’s 2026 high-risk AI requirements, and multiple other jurisdictions are developing frameworks that parallel the EU Act’s risk-based structure. Building governance infrastructure to meet EU AI Act requirements now effectively covers most of what other emerging regulations will require.

Here you can check red flags indicating unsanctioned AI usage in your organization. Check our latest risk classification guide for severity assessment.

A Practical Governance Checklist: Before, During, and After

Before deployment (governance infrastructure):

• Named Model Owner assigned for every deployed system
• Incident tier definitions documented and agreed
• Response team structure defined with named individuals
• Monitoring baseline established (input distributions, output distributions, business KPIs)
• Kill switch procedure documented and tested
• Legal review of any customer-facing AI outputs completed
• For high-risk systems: EU AI Act risk classification completed, logging infrastructure active

During an incident:

• Tier classification confirmed within the first hour
• Incident Commander engaged for Tier 2 and above
• Containment action taken before root cause investigation
• Incident documentation record opened immediately
• Legal lead notified for any incident with user-facing outputs
• External communication decision made (default: no public statement until facts confirmed, brief factual statement within 4 hours for Tier 3 events)

After the incident:

• Root cause analysis completed across all four layers
• Formal incident record completed and stored
• Process change or tooling improvement documented and scheduled
• Post-incident review with cross-functional team within one week
• If applicable: regulatory notification per EU AI Act or relevant requirements
• Monitoring thresholds updated based on learnings

Here you can check tiering methodologies for high-risk versus limited-risk AI systems. Check our latest ethics principles guide for practical implementation.

The Underlying Governance Philosophy

There’s a common mistake in how organizations approach AI governance: they build it reactively. An incident happens, lawyers get involved, a policy gets written, a committee gets formed. The policy and committee then describe what already happened, not what might happen next.

Governance built in reaction to incidents is governance that’s always one step behind.

The organizations that handle AI incidents well — that contain them quickly, communicate clearly, learn from them effectively, and face limited legal exposure — are the ones that built governance infrastructure before they needed it. They defined failure in advance. They assigned ownership before anything broke. They tested their kill switches before a production emergency. They documented their models before regulators asked.

NIST’s AI RMF uses the phrase “trustworthy AI” throughout — and the functional definition of trustworthy isn’t “AI that never fails.” It’s AI deployed within systems that have the governance infrastructure to detect failures, respond to them appropriately, and recover from them with integrity intact.

The 56% increase in documented AI incidents from 2023 to 2024 isn’t a reason to slow down AI deployment. It’s a reason to build the governance infrastructure that makes continued deployment responsible.

The incidents are coming. The question is whether you’re prepared when they arrive.

Here you can check actionable frameworks beyond theoretical guidelines for teams. Check our latest regulatory comparison for compliance alignment.