Every AI security vendor ships a benchmark chart these days. Most of them don't survive a second phone call.
On September 1, Cycode launched Agentic Code Scanning, a fourth layer that sits on top of its existing rule-based scanner and decides, code path by code path, whether a deterministic rule or a reasoning model should do the looking. The pitch lands at a useful moment: Lovable just raised $400 million at a $13.3 billion valuation, doubling in eight months, and AI-generated code is no longer a side project. It's shipping into production at companies that can least afford a bad merge.
Cycode's headline number: it ran the new engine against six real, published CVEs and caught all six, including two authorization flaws — a broken access control bug in gitea (CVE-2025-68941) and a middleware bypass in Next.js (CVE-2025-29927) — that no rule-based scanner in the test could catch, because there's no bad pattern to match. A missing permission check doesn't look like anything. Cycode's own deterministic scanner caught three of the six. Semgrep and CodeQL, tested separately, caught zero.
I asked Amir Kazemi, Cycode's senior director of product marketing, how the company picked its ten benchmark repositories — six pinned to CVEs, four deliberately vulnerable apps used as answer keys. He didn't have the specifics. "That's a question for our CTO," he said, and offered to follow up. That's not unusual on a launch call. It's also exactly the kind of question that decides whether a benchmark means anything.
Ronen Slavin, Cycode's CTO and co-founder, answered in writing a day later, and the detail went well past what made it into the press release.
The methodology, as he described it: the six CVE repos were the development set, used openly to build the engine. The four answer-key apps — vAPI, crAPI, DV RESTaurant, and Juice Shop, all existing open-source projects, not synthetic builds — were held out and untouched until testing. Before any fix went in, Slavin's team wrote down constraints in advance: no code change could reference a specific benchmark file, CVE ID, or line number, and any fix that moved the needle on the development set had to also hold up on the held-out apps. A hit only counted if it matched the actual mechanism behind the CVE, not just the file it lived in.
That structure exists to answer the obvious challenge: how do you know a reasoning model found the bug, instead of recalling it from a public CVE database sitting in its training data? Slavin's answer is specific rather than reassuring. The scanner is never told which CVE it's hunting, the test repos are pinned commits with no changelog or advisory text attached, and the findings are line-anchored data-flow traces written in the code's own variable names, not summaries that read like an advisory. On Jellyfin, he said, the model traced the vulnerable data flow across six function calls from an HTTP query parameter to the FFmpeg argument string, entering through a different controller than the one named in the public advisory. A model quoting a memorized answer doesn't take a six-hop detour to get there.
The press release's cleanest number undersold what the tool actually produced. It said a rule fired 573 times on mlflow, two hits landed in the CVE file, and Agentic Code Scanning returned "one validated, root-caused finding." Slavin's fuller account: on that same repository, the engine generated 87 raw candidate findings and published 38, each deduplicated and root-caused, including extra bugs the CVE never asked about — a symlink traversal in the artifact store, a zip-slip, and a template injection among them. The rule that fired 573 times, he clarified, belongs to Bearer — the same rule engine that already underlies Cycode's own deterministic scanner, which is a more interesting detail than the press release let on. The honest comparison, he argued, isn't 573 versus one. It's somewhere between 173 and 701 unranked hits per repository, depending on the repo, against 38 findings on mlflow that a developer could actually act on.
The single-run limitation in the published benchmark is real, and Slavin didn't dodge it. He called the published result "an existence proof, not a rate." Since the tested engine version shipped, gitea's authorization catch has reproduced in nine of nine repeat runs, and the full six-for-six result has held on a second model. Where a catch was inconsistent — Jellyfin missed on some runs until the team added a specific lead pointing to what reaches the vulnerable sink — Slavin says they found the cause and fixed it rather than rerunning until it worked. Cycode says it's now publishing catch rates across repeated runs going forward, not single screenshots, and has already extended its CVE corpus to 20 languages and 29 verified vulnerabilities.
None of this turns a vendor benchmark into independent research. It's still Cycode grading its own homework, and reasonable people will still want a third party to rerun it — something Cycode says is now possible with the corpus and configs it's publishing alongside the launch. But the gap between what showed up in the press release and what showed up under a direct follow-up is the real story here. As agentic security tools multiply — and Kazemi's own framing of a linear SDLC turning into a faster, circular ADLC suggests they will keep multiplying — the benchmark chart on the landing page is worth less than a vendor's willingness to answer the second question. Ask it before you buy.