Emerging AI Risk Categories: Novel Threats and Assessment Approaches

Beyond Established Risk Taxonomies

Current AI risk frameworks, including the EU AI Act, NIST AI RMF, and ISO/IEC 23894, were designed primarily around supervised machine learning systems with defined input-output relationships. The rapid evolution of generative AI, autonomous agents, and multi-modal systems has created risk categories that existing taxonomies capture imperfectly or not at all. Organisations performing AI risk assessments must supplement established frameworks with analysis of these emerging categories.

Emerging Risk Categories

Autonomous Agent Risks

AI agents that browse the web, execute code, manage files, and interact with APIs on behalf of users create risks that are qualitatively different from those of recommendation systems or classifiers. An autonomous agent can cause irreversible harm through a sequence of individually reasonable actions. The EU AI Act's human oversight requirements (Article 14) assume human-in-the-loop or human-on-the-loop designs, but autonomous agents may operate at speeds and complexities that make real-time oversight impractical.

Risk mitigation for autonomous agents requires sandboxing (limiting the actions agents can take), permission hierarchies (requiring explicit approval for high-impact actions), audit logging of all agent actions, and rollback capabilities for reversible actions.

Model Collapse and Synthetic Data Loops

As AI-generated content proliferates on the internet, new models trained on web-scraped data increasingly consume content generated by earlier models. Research published in Nature (Shumailov et al., 2024) demonstrates that iterative training on AI-generated data leads to model collapse: progressive degradation of output quality and diversity. This risk is systemic because it affects all models trained on common data sources.

Organisations should document the provenance of training data, filter AI-generated content from training datasets where possible, and monitor model performance for signs of capability degradation over successive training runs.

AI-Enabled Social Engineering

Generative AI dramatically reduces the cost and increases the quality of social engineering attacks. Personalised phishing emails, voice cloning for vishing attacks, and deepfake video for impersonation are now accessible to attackers with minimal technical skill. The EU AI Act Article 50(4) requires that deepfakes be labeled, but this obligation applies to legitimate users and is easily circumvented by malicious actors.

Assessment Approaches for Novel Risks

• Scenario analysis: develop specific threat scenarios for each emerging risk category rather than relying solely on historical incident data
• Red teaming: engage adversarial testing teams to probe AI systems for novel failure modes
• Horizon scanning: monitor academic research, incident databases, and threat intelligence for newly identified AI risks
• Cross-functional assessment: involve security, legal, ethics, and domain experts in AI risk identification workshops
• Dynamic risk registers: update AI risk registers quarterly rather than annually to capture rapidly evolving threats

Implications for Compliance

Organisations relying solely on EU AI Act classifications may underestimate their risk exposure. The Act's Annex III categories are static and were drafted before many of these risks were well understood. Proactive organisations should maintain a supplementary risk register covering emerging categories, even where regulatory obligations have not yet crystallised. ISO/IEC 42001 Clause 6.1 (actions to address risks and opportunities) provides the management system hook for incorporating emerging risks into established governance processes.

This article is for informational purposes only and does not constitute legal advice. Regulatory requirements change frequently — verify current rules with official sources. Built by Sawai Gyoseishoshi Office, Hiroshima, Japan.