Incident Response AI Style

October 2026

Author: Jim Routh

Before retiring six years ago, one of my favorite activities as an active CISO was leading synthetic cybersecurity incidents—practice sessions built around specific scenarios. Although these exercises are resource heavy, as they involve many operational leads and even the CEO, they are well worth the time and expense. The primary benefit comes from the wealth of learnings harvested during the event and the insight they provide into how decision‑makers behave under pressure. Exercises, like this, are resource and personnel intensive but well worth the time and expense in the long run. The primary benefit is the many learnings harvested during the event and the ability to see how decision makers prefer to make decisions during crunch time. 

One of the most valuable lessons learned is how 3^rd parties react to synthetic incident response events. Some of them are all over the prep work and anxious for an opportunity to shine in a made-up crisis. Others retreat behind a legal restriction or the famous, “it’s against our policy as a firm.” This is a very common response from cloud infrastructure as a service vendor. I had examples of cloud vendors claiming there was nothing they can learn from an incident because of the sophistication of their cyber resilience practices given that, “it’s cloud and therefore secure by definition.” I hope those days are behind us.

Major Takeaways:

1. I never participated in a synthetic incident without learning something significant and it typically centered around roles and responsibilities in incident response.  

2. Contract terms & conditions offer limited value in determining and influencing practical processes during an incident (real or synthetic).

3. I can think of dozens of instances where 3rd parties significantly out-performed the customer expectations and contract terms. Almost all these events were real incidents vs. synthetic.

Why does this matter now?

This is important today because enterprises are now starting to experience AI based cybersecurity incidents (the recent Hugging Face case incident, for example) where swarm agents were misbehaving and enterprise guardrails were either disconnected or not effective in containment. The basic fundamentals of cybersecurity incident response still apply to AI components gone rogue but there are additive measures required and different 3^rd parties at the table with limited incident response experience (eg: large model providers). In the case of the OpenAI/Hugging Face incident, a specialty firm (METR), was brought in to specifically study how the agents made decisions to form the Collective (their term) as a swarm and execute on their multiple attacks (Hugging Face and OpenAI file servers). The good news is that METR was instrumental in answering many questions about the thought/decision process and outcomes from the “Collective.” The bad news is how the swarm decided to “cover their tracks” by modifying transcripts and log files remains a bit of a mystery. 

Observability is essential for gathering facts in an incident response effort. Ensure that the observability capabilities include capturing transcripts to better understand why agents behave the way they do or did within the incident. This was key for OpenAI/METR.

OpenAI and Hugging Face sharing information that is invaluable to other enterprises has served crucial in understanding the limitations of guardrails, and decisions to turn them off, in real-world scenarios.

The recommended next step is to encourage the cybersecurity incident response team to schedule a synthetic incident where several of the company’s agents learned how to break out of the existing controls and attack other enterprises (like customers). Who are the players that should participate in that scenario (eg: limit the lawyers involved to save seats for operational and technical types)?  Which third parties should be included? How does an enterprise find a company like METR, who specializes in the analysis of agent decision making and rationales? Consider also inviting your cyber incident response firm that is on retainer to the exercise along with cloud providers. Ask your legal team to prepare a non-disclosure document specifically for the exercise if they wish (most MSAs are generally sufficient). 

The more frequently synthetic cyber incidents are exercised, the more predictable the outcomes will be and the more comfortable the enterprise will be working with third parties and knowing what practices are most effective for containment and applying the lessons learned. 

The good news is that both Anthropic and OpenAI have shared incident information with the public so there is a precedent. The bad news is with their respective levels of capitalization they can afford lawyers and communication experts that will instinctively limit participation the firm’s participation in customer cyber incident events. 

Consider establishing a standard engagement framework with your frontier model provider(s), cloud hosting providers and AI enabled SaaS vendors now before the complexity of AI-driven security incidents makes collaboration more challenging.

Jim Routh

Jim Routh is Chief Trust Officer at Saviynt and a Fellow at the Institute for Critical Infrastructure Technology (ICIT). A board member, advisor, investor, faculty member, and mentor, he previously served in senior cybersecurity leadership roles, including as CSO/CISO for American Express, DTCC, KPMG, Aetna, CVS, and MassMutual.

About ICIT

The Institute for Critical Infrastructure Technology (ICIT) is a nonprofit, nonpartisan, 501(c)3think tank with the mission of modernizing, securing, and making resilient critical infrastructure that provides for people’s foundational needs. ICIT takes no institutional positions on policy matters. Rather than advocate, ICIT is dedicated to being a resource for the organizations and communities that share our mission. By applying a people-centric lens to critical infrastructure research and decision making, our work ensures that modernization and security investments have a lasting, positive impact on society.  

Learn more at www.icitech.org.