Ask how much of the code shipping today is AI-authored and even the headline number turns out to be harder to define than it looks.
Veracode’s 2026 GenAI Code Security Report cites research from DX putting the figure at 51.9% 1. But the underlying DX analysis did not count generated lines inside repositories 2.
Instead, it used self-reported data from developers across more than 400 companies in Q2 2026. DX explicitly interprets the result as an estimate of how much coding workload developers delegate to AI tools, rather than a literal measurement of generated code volume.
GitClear, working from repository change data instead of a survey, puts AI-assisted commits closer to a quarter of the total 3.
The numbers are not really in conflict. They answer different questions, and that is a useful place to begin because the same compression happens throughout the AI coding debate. Generation speed becomes productivity, a passing test becomes correctness, a benchmark score becomes capability, and a model’s explanation becomes evidence that its implementation is sound.
The distinctions get lost faster than the numbers do.
AI systems can clearly generate useful code at increasing scope, but generation is not identical to developer productivity. Google’s 2025 DORA report studies software delivery performance across organizations and found higher AI adoption associated with higher throughput 4.
METR, an independent research nonprofit, examined a much narrower setting in its randomized controlled trial, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity 5. Sixteen experienced open-source developers worked on 246 real issues in repositories they already knew well. With the early-2025 AI tools tested there, they took 19% longer when AI use was allowed.
Neither result needs to cancel the other. They measure different layers of work.
The question worth asking in 2026 is therefore not simply how much code AI can produce. It is what happens after that code looks finished.
More specifically, can the surrounding engineering system distinguish generated output from output that has actually survived a check capable of proving it wrong?
The evidence below comes from maintainability data, delivery research, security testing, coding benchmarks, and a refactoring study that produces results less convenient for a simple anti-AI story.
These studies do not measure one common variable called “verification capacity.” They should not be forced into a single causal explanation.
What they keep running into is the same operational boundary:
Generated output and verified output are different states.
GitClear: the maintainability stress test
GitClear’s 2026 maintainability analysis covers 623 million code changes tracked from 2023 through mid-2026 3. Across that dataset, several structural indicators moved sharply.
Code block duplication increased 81% over the 2023 baseline. GitClear also tracks a proxy it calls moved code: lines relocated or restructured rather than newly written or deleted. That share fell from 21% of changed lines in 2022 to under 4% year to date in 2026.
Legacy refactoring, meaning cleanup of code untouched for more than a year, fell 74%. Functional connectivity fell 35%; GitClear uses that measure to track how often new code calls existing functions instead of reproducing their logic. Error-masking constructs, code that suppresses an error instead of addressing the underlying condition, increased 47%.
GitClear offers a plausible mechanism. AI coding systems are usually rewarded for satisfying the task immediately in front of them. Silently accommodating an unexpected value can therefore be an easier route to apparent completion than tracing why that value reached the function in the first place.
That explanation makes intuitive sense, but it is not what the dataset establishes.
The study observes rising AI-assisted development alongside large changes in repository structure. It does not isolate AI usage from time, repository composition, developer mix, changing workflows, or other tooling shifts.
The observed structural shifts are in the data. Assigning causal weight among those possible drivers is a separate problem.
That distinction matters because measures such as moved code and functional connectivity are proxies for maintainability, not maintainability itself. They can show that repository behavior is changing. They cannot, on their own, prove that one tool caused the change or that every repository experiences the same consequence.
DORA: output, productivity, and stability are different layers
Google’s 2025 DORA report drew on survey responses from nearly 5,000 technology professionals 4. Ninety percent reported using AI at work.
Higher AI adoption was associated with increased software delivery throughput. At the same time, it continued to have a negative relationship with delivery stability.
DORA describes AI as an amplifier. Strong feedback loops, testing systems, architecture, and delivery discipline can be accelerated, while weak ones can be exposed more quickly too.
More than 80% of respondents believed AI had increased their productivity, while 30% reported little or no trust in AI-generated code. Those results are not contradictory. They operate at different levels of the system.
This is also why the METR result belongs beside DORA rather than against it. DORA measures organizational relationships across a broad survey population. METR’s experiment measures task-completion time for experienced maintainers working inside repositories they already knew, under randomized AI/no-AI conditions 5.
The first found higher organizational throughput associated with greater AI adoption. The second found a slowdown in one specific and demanding development setting. METR itself cautions against generalizing the result to most software developers or other workflows.
Calling all of this “productivity” without specifying the layer erases the useful disagreement. Code generation, developer task completion, review burden, and organizational throughput are connected, but they are not interchangeable measurements.
Veracode: the security stress test
Veracode’s 2026 GenAI Code Security Report tested more than 100 models across security-relevant coding tasks 1. The average security pass rate was 56%, leaving 44% of evaluated outputs failing the relevant security requirement.
The aggregate conceals a wide spread. The best-performing model reached 68%. Reasoning models performed somewhat better than non-reasoning models, while model size itself made little difference.
Performance also varied sharply by vulnerability class. SQL injection tasks were handled far more successfully than cross-site scripting.
The cleaner historical comparison stays inside Veracode’s own series. Its earlier report recorded a security pass rate around 55%, while the 2026 result is 56%. That is a one-percentage-point movement across comparable measurements produced by the same organization.
A different study provides historical context without serving as a second point on the same trend line. Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions first appeared as a preprint in 2021 and was later published at the 2022 IEEE Symposium on Security and Privacy 6.
Pearce and colleagues tested GitHub Copilot across 89 security-relevant scenarios. Roughly 40% of the resulting programs were vulnerable.
The two studies use different benchmarks, protocols, and generations of systems. They should not be read as evidence that one vulnerability rate remained stable from one exact percentage to another.
What they show independently is narrower: materially different security evaluations both found substantial vulnerability rates in generated code.
Veracode’s current data supplies the more useful internal contrast. Syntactic correctness has become very high, while security performance remains uneven across vulnerability classes.
A solution that compiles and looks plausible is therefore weak evidence for the property that actually matters: whether the relevant security failure was tested and excluded.
SWE-bench: when the verifier needs verification
HumanEval is a widely used function-synthesis benchmark: given a description, produce an isolated function.
SWE-bench, introduced by Jimenez and colleagues, moved toward a harder setting by asking models to resolve actual GitHub issues inside real multi-file repositories 7. The original evaluation contained 2,294 software engineering problems drawn from 12 popular Python repositories. At launch, the best-performing model, Claude 2, solved only 1.96% of them.
Scores later rose dramatically on SWE-bench Verified, a curated version intended to address defects in the original benchmark. By early 2026, however, the evaluation itself had become the object of scrutiny.
In February 2026, OpenAI published Why SWE-bench Verified no longer measures frontier coding capabilities 8. The company audited 138 Verified tasks that its models frequently failed and found material test or problem-definition issues in 59.4% of them.
Some tests rejected functionally correct solutions because they required implementation details not stated in the task. Contamination analysis also found cases where frontier models could reproduce human-written reference fixes or problem-specific details from benchmark-related information.
OpenAI stopped reporting SWE-bench Verified and recommended SWE-Bench Pro instead.
A separate 2025 audit, The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason, reached the problem from another direction 9. Frontier models could identify buggy file paths from issue descriptions alone with accuracy as high as 76%, even without access to repository structure.
On tasks drawn from repositories outside SWE-bench, the best result fell to 53%. The difference is consistent with contamination or memorization contributing to benchmark performance.
The story did not stop with Verified.
Five months after recommending SWE-Bench Pro, OpenAI audited the replacement. In Separating signal from noise in coding evaluations, published July 8, 2026, the company examined Pro’s 731-task public split 10.
Its analysis pipeline classified 200 tasks, or 27.4%, as broken. A separate human annotation campaign identified 249, or 34.1%. OpenAI summarized the combined evidence as an estimate that roughly 30% of the public tasks were broken.
The failure modes matter because they go in both directions. Overly strict tests can reject functionally valid solutions, while underspecified or misleading prompts can make the intended target impossible to infer reliably. Low-coverage tests create the opposite problem by allowing incomplete fixes to pass.
That sequence matters more than the embarrassment of one benchmark replacing another. Verified was built to improve the original SWE-bench. Pro was then recommended when Verified stopped providing a trustworthy signal, only to fail a deeper audit itself.
Evaluation is itself an object of verification.
A grader does not establish model correctness merely by being called a benchmark, just as a green test suite does not establish product correctness merely by being green.
SWE-bench Pro: when the benchmark itself becomes the subject of evaluation
There is an opposite evaluation failure too.
Frontier Lag: A Bibliometric Audit of Capability Misrepresentation in Academic AI Evaluation examined applied-domain AI studies and found a substantial gap between the models evaluated in published research and the contemporaneous frontier 11.
The authors report that the median evaluated model lagged the frontier materially at evaluation time. Configuration details such as reasoning mode were also often poorly disclosed.
Benchmark failure can therefore distort apparent capability in either direction. Contamination and weak tests can make systems look better than the underlying capability warrants. Stale models and saturated benchmarks can make the frontier look weaker than it has become.
Long-context research supplies a narrower possible contributor to repository difficulty.
The 2024 TACL paper Lost in the Middle: How Language Models Use Long Contexts found that models performing multi-document question answering and synthetic retrieval often did better when relevant information appeared near the beginning or end of context 12. Performance tended to fall when that information appeared in the middle.
The study did not test software repositories, so it cannot establish the cause of SWE-bench failures. It does, however, make positional retrieval weakness a plausible mechanism worth testing rather than assuming away.
Repository-scale coding depends on recovering the helper, invariant, test assumption, or state transition that matters from a much larger body of context. A larger context window tells us how much information can be presented to a model. It does not, by itself, tell us whether the relevant constraint was actually used.
The benchmark story raises a question that matters just as much when a model performs well: what did the successful metric actually establish?
The refactoring study: a good score still has a contract
The 2026 ACM TOSEM study An Empirical Study on the Code Refactoring Capability of Large Language Models compared StarCoder2, GPT-4o, GPT-4o-mini, LLaMA 3, and DeepSeek-v3 with human developers [13]. The dataset covered 5,194 refactoring commits across 30 open-source Java projects.
GPT-4o and DeepSeek-v3 passed unit tests on more than 90% of multi-file refactorings within five attempts. LLaMA 3 produced the strongest code-smell reduction across much of the measured taxonomy.
StarCoder2 showed a narrower advantage on local transformations such as Extract Method and Inline Variable.
Humans did not simply win whenever a task looked complex. Their clearest advantages appeared in Rename Class and Encapsulate Attribute. Those refactorings depend more heavily on understanding what a class or field represents within the surrounding system.
That is a more precise result than saying models handle patterns while humans handle complexity. Some structural tasks that feel judgment-heavy become highly automatable once the objective is made explicit enough.
The study also shows why the objective matters. Higher code-smell reduction does not establish better refactoring overall. Developers may be preserving runtime behavior, architectural intent, long-term maintainability, naming conventions, or project-specific constraints that the smell metric does not capture.
SWE-bench and the refactoring study therefore appear to pull in opposite directions, but they expose the same measurement problem. A defective evaluation can mismeasure capability. A strong score on a valid metric can still support a much narrower claim than someone wants to make from it.
Before trusting either success or failure, you need to know what the check was capable of distinguishing.
Four states that should not be collapsed into “done”
That distinction becomes practical once “done” stops being treated as one state.
- Specified. The intended behavior is written down without claiming that the behavior already exists.
- Implemented. Code exists that performs the intended unit of work.
- Wired. The real product path actually reaches that implementation.
- Verified. The behavior has passed a predefined check capable of failing for the right reason, through the path the claim applies to, and the resulting evidence has been recorded.
The last state is stronger than execution. Running real input and recording an output proves that something happened. It does not prove that the observation discriminated between the intended implementation and a plausible wrong one.
A parser test, for example, should include malformed input that the implementation must reject. An authorization check should exercise a state that must not cross the boundary.
An HTTP integration claim should reach the actual network-facing path rather than a helper mocked underneath it. A packaging fix should be checked against the built artifact, not just the source tree.
The check has to be capable of failing for the defect it is supposed to catch.
That requirement also explains why a green result can coexist with a false product claim. A helper may be implemented correctly but never wired into the application, while a unit test validates the helper and bypasses the route the user actually calls.
A benchmark can fail in either direction as well. Hidden tests may reject good fixes because they demand undocumented behavior, or accept incomplete fixes because they never cover the requested feature.
These are different failures, but they share one mistake: assigning more authority to evidence than the evidence earned.
Verification should follow risk
The four states do not deserve equal ceremony on every change. A typo does not need an audit trail. A modification to an authorization boundary does.
For high-risk work involving security, authorization, external evidence, or public API contracts, the intended behavior should be explicit before implementation. The verification check should also be capable of failing on the specific defect the change is meant to prevent.
Ordinary feature work needs a narrower contract and at least one discriminating negative case. Formatting and local refactors usually do not justify the same cost.
This is not an argument for maximum testing. It is an argument for matching the strength of the verification claim to the consequence of being wrong.
SWE-Bench Pro is useful here precisely because the failure occurred one level above application code 10. The benchmark had tests, scores, and a public reputation as an improved evaluation surface.
That still did not establish that its tests were strict in the right places, permissive in the right places, or aligned with what its prompts actually asked models to do.
The same discipline therefore applies recursively. Code needs verification, and the mechanism granting verification needs evidence that it can discriminate correctly too.
Where this leaves the original question
Is AI-assisted coding a double-edged sword for software quality?
The metaphor works only if the edges are separated.
The generation advantage is increasingly visible. Models can produce useful code across larger scopes, and the refactoring study shows that some structural work once assumed to require deeper human judgment is already highly automatable under a clearly bounded objective [13].
The productivity effect is more conditional. DORA observes higher throughput alongside higher AI adoption at the organizational level 4. METR found a slowdown for experienced maintainers working on familiar repositories with early-2025 tools 5.
Both can be true because generation time, developer task time, review effort, integration cost, and organizational throughput are different parts of the work.
Quality is conditional in a different way. It depends on what evidence the engineering system requires before allowing a result to move from one state to the next.
Across the evidence above, the recurring problem was rarely the absence of a signal. There was usually something persuasive to point to: cleaner structure, higher throughput, a passing test, a strong benchmark score, or a successful refactoring metric. The harder question was whether that signal actually established the claim being made from it.
That is the practical use of separating Specified, Implemented, Wired, and Verified. Before calling a change finished, a team needs to know which state the available evidence actually supports. It also needs to know what failure the check was capable of exposing and what remains untested.
The distance between those answers is the verification gap this article has been describing.
References
- Veracode. 2026 GenAI Code Security Report: 100+ Models Tested. 2026.
- DX. AI-authored code has nearly doubled, but so has PR size. Engineering Enablement, June 17, 2026.
- GitClear. *The Maintainability Gap: AI Code Quality in 2026*. 2026.
- DORA / Google Cloud. State of AI-Assisted Software Development 2025. 2025.
- Becker, J., Rush, N., Barnes, B., Rein, D. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. METR, July 10, 2025.
- Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., Karri, R. Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions. IEEE Symposium on Security and Privacy, 2022.
- Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., Narasimhan, K.
. ICLR, 2024.
- OpenAI. Why SWE-bench Verified no longer measures frontier coding capabilities. February 23, 2026.
- Liang, S., Garg, S., Moghaddam, R. Z. The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason. arXiv:2506.12286, 2025.
- OpenAI. Separating signal from noise in coding evaluations. July 8, 2026.
- Gringras, D., Salahshoor, M. Frontier Lag: A Bibliometric Audit of Capability Misrepresentation in Academic AI Evaluation. arXiv:2605.04135, 2026. Preprint.
- Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., Liang, P. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 2024.
- Cordeiro, J., Noei, S., Zou, Y. An Empirical Study on the Code Refactoring Capability of Large Language Models. ACM Transactions on Software Engineering and Methodology, 2026.
