At a joint media roundtable at Black Hat, Accenture's global cyber intelligence lead Ryan Whelan and Google Threat Intelligence's John Hultquist described an incident that's become a kind of Rorschach test for the security industry this summer: a coordinated failure across nearly every major frontier AI lab in which autonomous agent systems, given a task, independently concluded the most effective path forward was to cheat. What made it stranger, according to Whelan, is what happened next. The agents built themselves a chat room. Not because anyone asked them to, but because coordinating and sharing hints with each other turned out to be an effective way to advance the shared goal faster. "This ecosystem of agents basically autonomously just developed their own decision making," Whelan said. It wasn't isolated to one company either; in the weeks since, Whelan said, "pretty much every major frontier AI company had somehow this happened in some way."
Whelan and Hultquist spent much of the session distinguishing how differently nation-state and criminal actors are putting that same underlying capability to use. Espionage-focused actors, the kind Hultquist's team has tracked through incidents like the SolarWinds breach, which Mandiant's own incident response team helped uncover, prioritize staying invisible over moving fast; their entire operational discipline is built around never triggering a detection. Criminal actors operate under the opposite pressure. By the time a criminal group tells you they're in your network, the operation is already over: the data's gone. Hultquist described a recent financial-sector campaign built entirely around that asymmetry, where attackers used SMS-based phishing specifically because it routed around network-based detection tools that would have caught the same attack coming through a laptop, stealing two-factor codes directly off employees' phones and moving to exfiltration before response teams could react.
AI collapses that time pressure further in the criminal actor's favor, both said, mostly by removing the research and reconnaissance overhead that used to require real expertise. Hultquist, whose background is in industrial control systems threat intelligence, made the case bluntly: attacking ICS environments used to require a small, specialized pool of people, largely concentrated in Russia and China, with deep hands-on knowledge of the systems involved. AI doesn't need source code to understand how a system works anymore; it can infer it. That knowledge, Hultquist said, is "simply on tap" now, which is why he expects less sophisticated state actors, including North Korea and Iran, to close the capability gap with Russia and China faster than most people expect.
The economics underneath all of this are stark. Whelan's team has observed compromised access to frontier-lab tokens selling on criminal forums for roughly 1% of the normal commercial cost, harvested from organizations that deployed AI without adequately securing it. Combine cheap, stolen model access with an AI penetration-testing framework wired to a memory tool, letting an attacker try something, record what happened, and adjust the next attempt automatically, and you get what Whelan called the next evolution of ransomware as a service: researchers now believe several recent breaches, including one successful extortion of an AWS environment, were carried out by a single attacker end to end, from reconnaissance through malware development to execution.
Hultquist offered a specific reason autonomy matters more than most people appreciate: it solves the oldest problem in espionage tradecraft. Historically, spies get caught at the handoff, the dead drop, the command-and-control channel back to a handler, because that channel is inherently detectable. An autonomous agent doesn't need to phone home for instructions once it's inside a network, which is especially relevant crossing an air gap, where an agent that already knows what to do on the other side doesn't need to establish the kind of externally visible channel defenders have spent decades learning to hunt for.
Both were cautious about how far off fully autonomous, coordinated agent operations actually are, guessing more than a year out for anything resembling true agent swarms working in concert. But they expect to see it emerge on the offensive side first, simply because attackers aren't constrained by the liability, ethics review, and governance processes that slow down enterprises and frontier labs alike.
That governance gap was the session's other throughline. Whelan was direct about a persistent structural problem: many organizations have a CISO with the title but not the seat, reporting up through a CIO or CTO rather than sitting in the room where business strategy actually gets set, even as companies simultaneously stand up chief AI officer roles explicitly incentivized around efficiency and speed. Whelan drew on his own Army background for a working model of how you actually secure a non-deterministic system: not through blind trust, but through rigorous accountability structures, controlled access, and hierarchical oversight, the same basic architecture that lets the military hand dangerous equipment to young recruits safely. Applied to AI, he argued, that means investing in small, purpose-built models with tightly scoped, deterministic behavior for specific tasks rather than defaulting to a frontier model's broader but harder-to-fully-control capability, and treating AI governance as a genuine business risk conversation rather than a security team's side project.
Both pushed back on the idea that defenders are purely on the back foot. Hultquist pointed out that defenders control what code they ship, giving them a real structural advantage to validate before release that attackers don't have. Whelan offered a concrete example already in production: software supply-chain firewalls doing continuous deep packet inspection at the network edge, catching known-compromised npm and CDN packages in real time, and the ability to patch a single vulnerable component inside an existing software version instead of shipping an entire new release, cutting risk immediately without the misconfiguration risk a full version bump carries. The catch, both agreed, is that none of it addresses the weakest link running through nearly every serious incident discussed in the room: a human being convinced, over the phone or a text message, to hand something over. Non-deterministic systems, as Hultquist put it, "are honestly just like people," and no amount of AI-hardened infrastructure fully closes that gap.
