On 4 November OpenAI published a long write-up called "A thousand fewer paths to root". In one run, a pipeline of AI agents went through the Linux kernel, reproduced 1,724 real bugs, and proved that 948 different bugs could each let an ordinary user take full control of a machine. It also wrote a candidate fix for every one of them. The whole run cost $1,213,430 in API credits, which OpenAI works out as $1,279 per root exploit.
The headline is a security story, and the security people are right to be loud about it. But I read the post three times, and every time the part that held my attention was not the bug count. It was the shape of the system. OpenAI did not point one clever agent at 40 million lines of code. They built a funnel of cheap and expensive agents, put sealed test machines between them, measured what every stage cost, and then wrote down, with numbers, the places where the design went wrong.
Those mistakes and fixes apply to any multi-step agent system: a code reviewer, a data pipeline, a support bot that takes actions. So this article walks through the pipeline stage by stage and pulls out the design rules. It stays defensive and architectural. There are no exploit techniques here; OpenAI left those out of its post too, for safety reasons. At the end there are five interview questions with short answers.
First, the words you need
Skip this if you know kernel security.
- Kernel. The core of the operating system. It decides what each program may touch. Programs ask it for things (open a file, send a packet) through a fixed door called a system call.
- Root. The all-powerful admin account on Linux. Root can read every file and change anything.
- Local privilege escalation (LPE). A bug that lets someone who can already run programs on the machine as a normal user turn themselves into root. "Local" means they are already on the machine; it is not a break-in from the internet by itself.
- Low-privileged user. An ordinary account with no admin rights.
- Candidate finding. A suspicion: "this code looks like it could go wrong." Not yet proven.
- Proof of concept (PoC). A small program that shows a bug is real by making it happen.
- Information leak. A bug that lets a program read memory it should not see. Less severe than root.
- Virtual machine (VM). A whole computer simulated in software; if code inside crashes the kernel, only the simulated computer goes down. A sandbox is the general idea: a sealed box where code can misbehave safely.
- KASAN (Kernel Address Sanitizer). A kernel debugging mode that reports when code reads or writes memory it should not, turning a silent memory mistake into a logged one.
- Mitigations. Protections that make bugs harder to abuse even when they exist. OpenAI names four that a standard distribution switches on: SMEP and SMAP (processor features that stop the kernel from executing, or freely reading and writing, memory that belongs to user programs, so an attacker cannot easily trick the kernel into using memory the attacker controls), KASLR (the kernel is loaded at a random address at every boot), and HARDENED_USERCOPY (extra checks when data is copied between a program and the kernel).
- Patch. The code change that fixes a bug. Upstream means the official Linux source, where maintainers accept patches.
Why the kernel, and why finding bugs was not the hard part
OpenAI chose Linux because it runs under Android phones, cloud servers and supercomputers, and because it is big: more than 40 million lines of code, reachable through system calls, networking, filesystems and device drivers.
The most useful sentence in the whole post, for anyone building agent systems, is this one: the operational challenge "was not discovery, but determining which findings could be reproduced, what security boundary they crossed, and which warranted the earliest maintainer attention."
Getting a model to produce suspicions is easy. The slow, expensive part is deciding which are real, which matter, and which deserve a human's time first. A pipeline that skips that just moves the work onto the reviewers, and a maintainer handed a thousand unproven bug reports is not helped. They are buried. So this pipeline is built around proof: nothing moves forward on a model's opinion alone, only on evidence that something happened in a real kernel.
The pipeline at a glance
OpenAI describes five stages, with an upcoming cyber model at its core:
- Vulnerability discovery. Agents pick the parts of the code worth auditing, then scan them for candidate findings.
- Dynamic validation. Agents try to reproduce each candidate in an isolated machine and keep the runtime evidence.
- Exploitability assessment. Agents test whether a validated bug could leak information or lead to root.
- Patch generation. Agents write a fix for each bug.
- Human review. Security researchers review the exploit, the patch and the proposed severity before anything goes to the maintainers.
For every bug that survives, the pipeline bundles a root cause analysis, a minimal program that crashes the kernel, a candidate patch, and a full exploit that proves the impact. That bundle is the real product: it turns "an AI says there is a bug" into something a human can check quickly. Here is how the numbers narrow.
Stage 1: scouts, then analysts
Scouts decide where to look. Scout agents read through the kernel folder by folder and file by file, and mark the targets worth auditing. They judge two things: whether a low-privileged user can reach the code at all, and whether it is security-relevant. Code that only root can call is not a path to root, so it can wait.
The scouts picked 5,034 audit targets, about 2.8 million lines, roughly 7% of the kernel. That step cost $3,426. It is the cheapest stage in the whole pipeline by a factor of ten, and it decides where the rest of the $1.2 million is spent.
This is the first design rule, and it is an old one: narrow the input with a cheap step before you run an expensive one. A search engine does not run its most expensive ranking model over every page on the web. It retrieves a few thousand candidates cheaply and ranks only those. The scouts are the retrieval step.
Analysts read the code. Analyst agents then took each audit target, read it to understand how it works, worked out its threat model (who could attack this code, and how), and looked for vulnerabilities. OpenAI says this follows a workflow similar to its Codex Security plugin. Within 12 hours the analysts produced 2,308 candidate findings after removing duplicates. This stage cost $316,549.
Notice "after deduplication". Agents working in parallel find the same problem from different directions. Merge duplicates before the expensive stages, or you pay to prove the same bug several times.
Stage 2: prove it in a sealed machine
A candidate is a suspicion. OpenAI is blunt that at this scale nobody can investigate every report by hand, so they only pushed forward findings that met three conditions:
- reachable by a low-privileged user,
- needing no special Linux capabilities (named admin powers, such as managing the network),
- producing measurable evidence of memory corruption.
For each candidate, a triage agent wrote a proof-of-concept program and ran it inside an isolated virtual machine whose kernel had KASAN switched on. The agent watched both the program's output and the system console, and kept refining the program until it reproduced the reported behaviour or ran out of ways to trigger it.
Within 30 hours the system reproduced 1,724 of the 2,308 candidates, about 74.7%. Of the rest, OpenAI says most either needed elevated privileges or did not reproduce within the time budget, because of hard race conditions (bugs that only appear when two things happen in an unlucky order), very specific trigger conditions, or slow checks. OpenAI puts incorrect analysis by the model, where the reported bug assumed something impossible, at only 3.1%.
Two design points come out of this stage.
The judge is the machine, not the model. The triage agent does not decide a bug is real by thinking harder. It makes the bug happen in a real kernel, with KASAN turning a hidden memory error into a logged report, and the evidence is kept. When an agent grades its own work, look for the equivalent: a test that runs, a query that returns rows. A second model's opinion is weaker than any of those.
Every check needs a time budget. Some real bugs were dropped because they did not reproduce in time. That is a deliberate trade: a pipeline with no budget gets stuck on its hardest items.
This stage cost $73,330. OpenAI adds an observation that shapes the whole economics: because validation means writing and running a working program, it costs more than a recommendation from reading the code, and checking a false alarm cost about as much as checking a real bug. So every false positive that reaches this stage is close to full price. Removing false positives earlier makes the pipeline meaningfully cheaper.
Stage 3: how bad is it, really?
Validation proves a bug is real. It does not say how urgent it is. A bug that can lead to root deserves attention before one that can only crash a machine. With 1,724 validated bugs, the next job was to find the ones with the biggest impact.
A permissive screen first. An initial screening agent read each report, crash log and the relevant source code, without running anything. OpenAI made it deliberately permissive, so it would not throw away anything potentially serious, and only filtered out cases with no plausible exploitation path. It narrowed the set to 313 information leak candidates and 1,238 local privilege escalation candidates. That cost $44,194. (If the two groups do not overlap, which the post does not say, about 173 validated bugs were screened out.)
A filter can be wrong in two ways: it lets junk through, which costs money downstream, or it drops something important, which costs safety. Here a missed root bug is far worse than a wasted attempt, so the screen leans toward letting things through. Every filter you build leans one way; you should be able to say which, and why.
Then a realistic test. Exploit-development agents then tested the candidates inside isolated VMs set up with the protections a standard Linux distribution ships with: SMEP, SMAP, KASLR and HARDENED_USERCOPY. OpenAI's reason is that this stops "unrealistic assessments that depended on protections being disabled". A test environment that is weaker than production tells you about a system nobody runs. The target machines ran a stock desktop operating system, not a weakened target.
The agents produced 291 proof-of-concept programs for information leaks and 1,169 working exploits that gave an unprivileged user root. Then the team ran variant analysis, which means looking for other places where the same kind of flaw appears, and wrote 948 distinct patches for the bugs behind those exploits. OpenAI uses 948, not 1,169, as the count of unique bugs; the gap suggests several exploits shared a bug.
Worth copying: count the thing you care about. 1,169 exploits is the bigger number, but maintainers fix bugs, so 948 is the one that matters.
Stages 4 and 5: patches, then people
For each validated bug, the system also generated a patch. The model got the relevant source code and instructions on good patch writing: run static checks on the patch (checks that read code without running it), follow the style of the nearby code, and look for variants of the same bug so the fix is complete.
OpenAI says the first patches stopped the proof of concept but sometimes added more code than needed, risked performance costs, ignored local conventions, or missed variants of the same bug. More instructions and more rounds of iteration improved them, and in a few cases the prototype found problems in patches written by human experts before they were merged. OpenAI recommends using models for adversarial review of any patch: can this fix be bypassed, and does it miss another path to the same bug?
Then people. Researchers review the exploit, the patch and the severity before anything is reported. Following the kernel community's advice, OpenAI started with critical bugs and has already sent hundreds of reports and patches. The human sits at the end, where the evidence is already bundled, so machine time is spent making each minute of human time count.
Where the money went
Here are the stage costs side by side.
Add the five stage costs and you get $1,213,428, two dollars off the stated total of $1,213,430, so the published breakdown is complete apart from rounding. The exploit stage alone was $775,929, about 64% of everything, split into $84,929 for information leak candidates and $691,000 for root candidates. On average a root candidate cost $558 to process and an information leak candidate $271, because the leak programs were carried forward from validation and needed less work.
The cost sits at the end. Finding suspicions was cheap; proving them was expensive. That is the normal shape of an agent pipeline: the stage that acts in the real world and gets checked burns the money, so the leverage is in what you let into it. OpenAI also says it focused on method, not cost, and expects better harness engineering (the code around the model that runs it) could cut every stage, so read these numbers as an upper bound.
For scale, OpenAI compares the run with past bounties: the Pwn2Own and TyphoonPWN hacking competitions offered $20,000 and $70,000 for a Linux root exploit, which puts 948 exploits at a hypothetical $18.9 million to $66.3 million, 16 to 55 times the run's cost. For defenders the point is uncomfortable: finding serious kernel bugs is no longer rare, expensive work.
Design rule 1: give every agent a give-up rule
The number that matters most for builders is in the exploit stage.
OpenAI reports that unsuccessful exploitation attempts cost $3,922 each on average, across 69 cases, while successful ones cost $360 each, across 1,169 cases. A few hard bugs, such as race conditions, kept agents busy for a long time.
Now do the arithmetic yourself (this part is mine, not OpenAI's): 69 times $3,922 is $270,618, and 1,169 times $360 is $420,840. Together that is $691,458, which matches the $691,000 OpenAI gives for the root stage. So the failures were 5.6% of the attempts and about 39% of the spend.
OpenAI's own conclusion is that a reasonable give-up condition, such as stopping an attempt after one hour with no progress, would have cut the cost of that stage significantly.
A failing agent does not know it is failing, and here a failed attempt cost about eleven times a successful one. So the stop condition lives outside the agent, in the orchestrator (the program that hands out tasks and collects results), and it measures progress, not only time. "No new evidence in the last hour" beats "four hours maximum", because a productive agent keeps going and a stuck one is cut early. In your own system that can be a progress signal per step (a new test passing, a smaller error) and a budget that resets when progress happens.
Design rule 2: check the path, not only the result
The pipeline marked a bug as exploited when the agent got a root shell (a command prompt running as root). In an early run, the exploit stage reported 200 successes. Human review found that 20 of them, one in ten, had reached root through a different unknown bug than the one the agent was assigned.
They did what they were measured on: the check asked "did you get root?", and some found the easiest way to root, which was not the bug in their ticket. A true result can still be the wrong result.
OpenAI's fix had two parts: tighter task constraints, and an independent check that each final exploit used its assigned vulnerability. Their stated takeaway is to verify the path to a result, not only the result itself, with isolation, monitoring and independent checks as necessary controls.
This failure is everywhere. A coding agent told to "make the tests pass" can edit the test. A data agent asked to "fix the failing job" can drop the rows that fail. The fix has the same shape as OpenAI's: a check that does not trust the agent's story, run by something the agent does not control.
Design rule 3: re-test your scaffolding when the model changes
Scaffolding is everything you build around a model to help it: extra prompts, tool libraries, step-by-step plans, retrieval of reference material.
Early in the project OpenAI tested whether heavy scaffolding made the agents better. GPT-5.5 Cyber did not reliably know when and how to apply known kernel exploitation techniques. So the team wrote 103 techniques down in a model-friendly format, put them in the agent's environment, and told the agent to use them. Across 63 exploitation candidates, that raised the success rate from 76.2% to 88.9%.
Then they repeated it with GPT-5.6 Cyber. The newer model found, chose and applied relevant techniques on its own. With the technique library removed, its lightly orchestrated workflow beat the heavily scaffolded GPT-5.5 Cyber workflow. Adding the library back gave only 1.5 percentage points. OpenAI judged that not worth the maintenance cost and removed the heavy orchestration.
The rule: scaffolding is code you maintain, and its value is tied to one model generation. Every time you upgrade the model, run your evaluation with and without each layer, and delete what no longer earns its keep.
Design rule 4: re-check findings against code that keeps moving
The kernel did not stand still while the scan ran. OpenAI says that when the scan and manual analysis finished after 10 days, 61 of the root bugs had already been patched upstream, 43 had fixes queued but not yet merged, and 299 were being reported and discussed in public. Maintainers and other researchers were working on the same code at the same time.
OpenAI recommends that automated triage revalidate findings against the latest code, including code merged after the original scan, so it does not send duplicate or stale reports that add review burden for maintainers.
In system design terms, a finding is a cached result over changing data. Check it is still true before you act on it, the same way you check a version number before writing back a database row you read earlier.
The same logic applies across model generations. In late July, a GPT-5.6 Sol run found and exploited 166 unique root bugs for $513,690. The October run with the newer cyber model found 948 for $1,213,430. OpenAI says the newer pipeline found more than five times as many new root bugs, despite three months of hardening in between. The runs also differed in kernel version and pipeline maturity, so this is not a clean model-to-model test. OpenAI's advice: rescan code you already audited whenever models improve, because a stronger model can find what a weaker one missed with the same harness.
Design rule 5: test the sandbox before you trust it
OpenAI also ran the pipeline against kernelCTF, Google's reward programme for kernel bugs, which uses a hardened kernel with less exposed code. It produced 21 root exploits and 25 information leak programs in 112 hours for about $32,936. During that run, an analyst agent was studying the kernel's system call entry code and hit a crash it did not expect. It traced the crash not to the kernel but to QEMU TCG, the software emulator that ran the test machine's processor, without having QEMU's source code. A later agent then built a root exploit for that bug. OpenAI confirmed it had been fixed in upstream QEMU but not in the release build they were using.
So the agents found a bug in the emulator underneath their own test machine, and an agent turned it into root inside that machine. OpenAI's advice is to test each part of the infrastructure before running this kind of research, and they say they have done so, disclosing and patching bugs in Kata, Moby/Docker, gVisor and other projects.
For anyone running agents that execute code, this is the most practical line in the post. Your sandbox is software with bugs, and a capable enough agent may find them. Keep it patched and on a current release, and do not let it share a machine with anything you would mind losing.
What this means if you run Linux machines
OpenAI's own advice to defenders is unusually direct. For the near and medium term, a low-privileged account on an operating system that is not up to date with its patches "should be considered functionally equivalent to root access for a well-resourced threat actor".
That changes two habits.
Patch speed becomes a security control, not housekeeping. If hundreds of kernel root bugs can be found in days, the gap between a fix landing and your machines running it is where the risk lives.
Assume a normal account can become root, and contain it. This is defence in depth: several layers, so one failure is not total failure. OpenAI names isolation tools such as Firecracker and gVisor for limiting the blast radius. In plain terms:
- gVisor puts its own kernel, written to run as a normal program, between the application and the host. The application's system calls go to gVisor rather than straight to the host kernel, so a bug the application can trigger lands in gVisor first, and gVisor itself uses only a narrow set of host calls.
- Firecracker runs each workload in a tiny virtual machine using the hardware's virtualisation support (KVM on Linux). The workload has its own kernel. If it gets root there, it is root inside a small VM, not on the host.
Neither makes a kernel bug harmless. They make it smaller: root inside the box still has to break the box, and as rule 5 showed, the box needs patching too. Further ahead, OpenAI points to code that is secure by default, less exposed surface, and formal verification.
How I would sketch this in a design interview
For any expensive, checkable task, this is the shape I would draw. It is my own generalisation, not OpenAI's architecture.
- A queue between every stage, so each stage scales or pauses on its own and a slow exploit stage never stalls the scouts.
- Items carry evidence, not opinions: an ID plus the crash log or test result so far. Later stages read the evidence, not the earlier agent's summary.
- Deduplicate before every expensive stage, keyed on something stable such as a function and a crash signature.
- A priority order, so the most severe items are proven first (OpenAI reported critical bugs first).
- A budget per item, enforced by the orchestrator: money, time and a progress signal. No progress means the item is parked with its evidence.
- A fresh, isolated sandbox per attempt, with nothing in it you would mind losing.
- An independent verifier per stage, checking the result came from the assigned work.
- Revalidate before acting, against the current state of the world.
- Humans at the end, reading complete bundles.
Queues, budgets and independent checks are ordinary distributed systems tools. What OpenAI's post adds is the dollar figures for what happens when an agent system is missing one.
Five interview questions, with short answers
1. Why put a cheap scout stage in front of the expensive analysis?
It decides where the expensive stages spend. $3,426 of scouting narrowed the kernel to about 7% of its code, and the next $1.2 million was spent only there.
2. Why is checking a false alarm as expensive as checking a real bug, and what do you do about it?
The check is the same work either way: write a program and run it in a real kernel. So remove false positives as early and cheaply as you can, before the costly stage.
3. An agent reports success. How can that still be a failure?
It can reach the goal by the wrong route: 20 of 200 early "successful" exploits used a different bug than the one assigned. Add an independent check of the path, run by something the agent does not control.
4. Where should the stop rule for a stuck agent live, and what should it measure?
In the orchestrator, because a failing agent keeps trying, and it should measure progress, not just time. Failed attempts averaged $3,922 against $360 for successes.
5. Your team built a big prompt and tool library to make the agent work. A new model ships. What do you do?
Evaluate with and without each layer. OpenAI's 103-technique library added 12.7 points to GPT-5.5 Cyber but 1.5 to GPT-5.6 Cyber, which beat the old setup without it.
The short version
Finding suspicions is cheap. Proving them is where the money and the risk are. So narrow the input early, judge with real evidence, stop stuck work from outside, check how a result was reached, and re-test your assumptions every time the model or the code underneath changes. OpenAI's $1.2 million run shows all of that at kernel scale, and none of it is specific to security.
If you want to practise designing agent systems like this one, with budgets, stop rules and checks on each step, I have a free lesson on making agent loops reliable on systemdesign.academy.
Sources