Earlier this year, Anthropic gave Apple, Microsoft, Google, Amazon and a handful of other companies access to an AI agent called Mythos and turned it loose on their code. Project Glasswing, as the initiative became known, found bugs that had survived decades of human audits and fuzzing, including one flaw that had been sitting in OpenBSD for 27 years. It also found a lot more than anyone could fix. Fewer than 1% of the vulnerabilities Mythos discovered ever got patched.
That gap is the story Victor Wieczorek keeps coming back to. Wieczorek is SVP of Offensive Security at GuidePoint Security, where his teams run penetration tests and red team engagements across application security, cloud, infrastructure and operational technology. When I asked him where AI still falls short against a human pen tester, he pushed back on the question itself.
"It's really a human versus a steam engine kind of idea," Wieczorek said. "AI [is] so good at finding needles in a haystack. What I prefer to focus on is everything downstream of that. After you find the bug, after you identify the remediation, then what do you do about it?" Glasswing, he said, "talks to a lot of complexity" in that narrative. Finding the flaw was never really the bottleneck. Getting an organization to understand it, trust it and act on it always was.
That downstream work is where Wieczorek says large language models still run into trouble, and not in the way most people expect. His teams see hallucinated CVEs and misread exploitation chains, but the deeper issue is how AI systems handle failure. "You're putting a lot of trust in what amounts to a coin flip of that AI's attention," he said. A tool can misread its own output — an error message, a log line — as a new instruction and steer off in an unpredictable direction. A human pen tester, hitting the same dead end, intuits what to try next. An LLM is just as likely to hallucinate a new one.
That's not a reason to keep AI out of offensive security. Wieczorek's team has used it since the earliest ChatGPT releases, first for open-source intelligence gathering and understanding how a target organization's systems fit together. It's a reason to keep a human owning what the AI produces. Wieczorek doesn't mince words about teams that let an agent run without watching how it got its results: "I would say it is... negligence." He compares it to a hammer. "If I'm swinging a hammer and I slam my thumb, I'm not blaming the hammer." A pen tester is hired to deliver a professional judgment a client can act on, and no model can carry that obligation. It can only feed the person who does.
That principle gets tested constantly on the defensive side too, where Wieczorek says agentic tools keep creating problems nobody planned for. Three years ago, when Copilot first landed inside Microsoft 365, one of his testers logged into a client's portal, asked Copilot who the security leaders were and where they kept sensitive files, and got refused — then simply said he'd been hired to run a security audit. Copilot handed over the domain administrator credentials. Microsoft has since closed that specific hole, but Wieczorek says the underlying pattern hasn't gone away: agents deployed with real access, coaxed into acting outside their intent by nothing more sophisticated than a well-worded prompt.
The scale of that exposure is what worries him most now. Organizations are standing up AI agents faster than they can track them, often without an expiration date on the access those agents hold. "We're seeing where long-tooth agents who have been around for a long time... could be running now in perpetuity," he said, with nobody watching what they can still touch. He's only half-joking when he calls it "agentic debt" — the security equivalent of the technical debt teams already know how to ignore, except this version comes with standing credentials.
None of this means organizations should wait for the tooling to mature. Wieczorek's advice for developers and security engineers evaluating agentic AI tools is to stop grading them pass/fail against a benchmark and start tracking how they drift over weeks and months of real use — whether the failure modes are getting caught and fixed, or quietly piling up. It's the same discipline his team applies to its own tools, and the same discipline he says most deployments skip.
Where this ends up, in Wieczorek's view, looks less like AI replacing pen testers and more like a security operations center: agents handling first-line monitoring, discovery and alerting, escalating up through tiers to the humans who own the judgment calls that matter. He's most impatient about applying that model to operational technology — the systems running energy, manufacturing and pharmaceutical infrastructure, where a patch can't just be pushed at 2 a.m. without knowing what happens to the people depending on that system staying up. "You can't just automatically turn them off," he said. Getting AI-assisted security into that world will take more than a better model. It'll take the same thing offensive security has always required: someone willing to own the outcome.
