Key Definitions
| Term | Definition |
|---|---|
| AI Incident | An event involving an AI system that results in or has the potential to result in harm to individuals, organizations, or society |
| Serious Incident | An incident or malfunctioning of an AI system that directly or indirectly leads to death, serious damage to health, serious disruption to critical infrastructure, or serious breach of fundamental rights (EU AI Act Art.3(49)) |
| Near Miss | An event that could have resulted in an AI incident but did not, providing learning opportunity |
| Incident Response | The organized approach to addressing and managing an AI incident, from detection through recovery |
| Containment | Actions taken to limit the scope and impact of an active AI incident |
| Root Cause Analysis | A systematic investigation to identify the fundamental cause(s) of an incident |
| Remediation | Actions taken to correct the conditions that caused or contributed to an incident |
| Post-Market Monitoring | Ongoing surveillance activities to detect incidents and performance issues after AI system deployment (EU AI Act Art.72) |
| Market Surveillance Authority | The national authority responsible for enforcing the EU AI Act in a Member State |
| Corrective Action | Action to eliminate the cause of a detected nonconformity and prevent recurrence |
| Incident Commander | The individual responsible for leading the incident response effort |
| Runbook | A documented set of procedures for handling specific types of incidents |
Chapter 1: Foundations of AI Incident Response
AI incident response is the structured process of detecting, assessing, containing, investigating, and recovering from events where AI systems cause or threaten to cause harm. Unlike traditional IT incident response that primarily addresses system availability and data security, AI incident response must also address harm from AI outputs — discriminatory decisions, dangerous recommendations, privacy violations through inference, and cascading effects from automated processes operating at scale. The EU AI Act formalizes incident reporting obligations for serious incidents, making AI-specific incident response a regulatory requirement.
1.1 Why AI Incidents Are Different
AI incidents differ from traditional IT incidents in several fundamental ways:
| Characteristic | Traditional IT Incident | AI Incident |
|---|---|---|
| Detection | Often clear (system down, error message) | May be subtle (gradual drift, hidden bias) |
| Scope | Usually confined to system boundary | May cascade through downstream decisions |
| Root Cause | Typically a specific failure point | Often systemic (data, model, process interaction) |
| Impact | Usually measurable (downtime, data loss) | May be diffuse and delayed (discriminatory patterns) |
| Reversibility | Often reversible (restore backup) | AI decisions may be irreversible (denial of services, harm) |
| Scale | Limited by system capacity | AI can affect thousands before detection |
| Attribution | Clear system/component failure | Difficult to attribute between data, model, and context |
| Remediation | Fix the component, restore service | May require model retraining, data correction, process redesign |
1.2 AI Incident Taxonomy
Category 1: Performance Incidents
- Model accuracy degradation below acceptable thresholds
- Increased error rates across all or specific populations
- System producing unreliable or inconsistent outputs
- Model drift causing progressive performance decline
Category 2: Fairness and Discrimination Incidents
- Discriminatory outcomes identified across protected characteristics
- Disparate impact exceeding acceptable thresholds
- Systematic disadvantage to vulnerable populations
- Proxy discrimination through correlated features
Category 3: Safety Incidents
- AI decisions leading to physical harm or danger
- Failure of safety-critical AI controls
- AI system operating outside safe operational boundaries
- Incorrect safety recommendations or alerts
Category 4: Privacy Incidents
- Unauthorized disclosure of personal data through AI outputs
- Model memorization exposing training data
- Inference attacks revealing sensitive information
- Privacy violations through profiling or behavioral prediction
Category 5: Security Incidents
- Adversarial attacks on AI models
- Data poisoning affecting model behavior
- Model theft or extraction
- Prompt injection or manipulation
- Unauthorized access to AI systems or training data
Category 6: Transparency and Trust Incidents
- AI system providing misleading explanations
- Failure to disclose AI involvement in decisions
- Misrepresentation of AI capabilities or limitations
- Loss of stakeholder trust due to AI behavior
Category 7: Operational Incidents
- AI system unavailability affecting critical processes
- Integration failures between AI and operational systems
- Unintended AI behavior following system updates
- Resource exhaustion or capacity issues
1.3 Severity Classification
| Severity | Definition | Response Time | Escalation |
|---|---|---|---|
| Critical (S1) | Serious incident per Art.62: death, serious health damage, serious infrastructure disruption, or serious breach of fundamental rights | Immediate (within 1 hour) | Executive team, regulatory authority |
| High (S2) | Significant harm to individuals or organization; regulatory compliance at risk; widespread impact | Within 4 hours | Senior management, legal team |
| Medium (S3) | Moderate impact; limited scope; no immediate harm but requires prompt attention | Within 8 hours | AI governance team, system owner |
| Low (S4) | Minor impact; contained scope; no harm but requires investigation and corrective action | Within 24 hours | AI operations team |
| Informational (S5) | Near miss or anomaly; no impact but valuable for learning | Within 5 business days | AI governance team |
1.4 Regulatory Reporting Obligations
EU AI Act Art.62 — Serious Incident Reporting:
Providers of high-risk AI systems placed on the EU market must report serious incidents to the market surveillance authority of the Member State where the incident occurred.
Reporting Trigger: Any incident that directly or indirectly leads to:
- Death of a person
- Serious damage to the health of a person
- Serious and irreversible disruption of the management and operation of critical infrastructure
- Breach of obligations under Union law intended to protect fundamental rights
- Serious damage to property or the environment
Reporting Timeline:
- Immediately after the provider establishes a causal link (or reasonable likelihood)
- No later than 15 days after the provider becomes aware of the serious incident
Report Content:
- Description of the AI system and its identification
- Description of the incident and circumstances
- Information about affected individuals (without personal identification)
- Measures taken or planned
- Any other relevant information