The Slop Gap: Why Rising Model Capability Hasn't Solved Code Quality And what happens when you turn “slop” into a falsifiable engineering hypothesis
The gap nobody's benchmark shows
Model capability on coding tasks has risen sharply. The useful question is not whether those scores are high; it is what they actually measure.
On February 23, 2026, OpenAI disclosed flaws in SWE-bench Verified, including task-specific information that let some models reproduce gold-patch behavior. Its audit found material test or issue-description problems in at least 59.4 percent of the 138 tasks it reviewed.
The contemporaneous top-line movement, 74.9 to 80.9 percent, is therefore not a clean measure of general autonomous engineering ability. It is evidence that benchmark numbers need provenance, split, scaffold, and budget attached before they can carry much interpretive weight. 1
Contamination-resistant evaluations such as SWE-bench Pro report materially lower performance, but aggregating their results into a single 23-to-59-percent range would hide the conditions that matter: the model, public or private split, agent scaffold, and reasoning budget.
That contamination-resistant label is now itself in question: days before this piece went out, OpenAI ran the same kind of audit on SWE-bench Pro and found roughly 30 percent of its tasks broken in ways that misrepresent model capability, retracting its own recommendation that the field treat Pro as the leading coding eval. This is the second major coding benchmark OpenAI has walked back in five months. 2
A related distinction is even more important for this note. GBQA asks an agent to discover a bug rather than repair an already named issue; its best reported setting found only 48.39 percent of verified bugs. Repairing a known defect and finding an unnamed one are different capabilities. 3
That is the gap this note is about. Generation capability and detection capability are not the same curve, and right now they are not even close.
Why slop keeps shipping while models keep improving
The persistence of slop is not evidence that models are simply weak. It is evidence that generation and verification scale differently.
- Review is the bottleneck.
Faros AI's observational study of more than 10,000 developers across 1,255 teams associated high AI adoption with more completed work and merged PRs, but also more review time, larger PRs, and more bugs per developer. Faster generation can move the queue into review rather than create trustworthy throughput. 4
- Local success is not architectural judgment.
Current coding agents are generally more reliable on local, well-scoped edits than on cross-cutting decisions involving dependencies, historical conventions, and architectural intent (Imai, 2022). 5
- Repository context is more than file volume.
Even when a repository fits within a large context window, an agent may not recover its implicit conventions and task-relevant dependencies reliably. A locally correct change can still violate a poorly represented boundary. 4
- Supply-chain plausibility is still risky.
In a 2026 package-hallucination replication, 53 of 127 package names hallucinated by all five tested models remained registerable. That supports separate existence checks; it does not validate every dependency-quality metric. 6
The result is familiar to maintainers: code can look plausible enough to pass a hurried review while still creating structural work for the next engineer.
What "slop" means here
The term has a broader social meaning than this tool can measure. "Why Slop Matters" describes a family resemblance including superficial competence, asymmetric effort, and mass producibility. 7
For this detector, we operationalize only the first: the mismatch between superficial competence and underlying substance. The other two remain central to the ecosystem-level maintainer problem, but are outside static analysis.
That distinction matters. Open-source maintainers have reported AI-generated contributions consuming review capacity without producing valid fixes 8, while developer-LLM studies document automation bias and related review risks. 9 Those are reasons to improve inspection; they are not evidence that one score has measured all of “AI slop.”
Scope: a review-risk hypothesis, not a definition of AI slop
The detector does not measure AI authorship, developer intent, or software quality as a whole. It operationalizes a narrower hypothesis: some observable structural mismatches are useful, auditable risk signals for review prioritization.
The indicators are adjusted through repository-local operational feedback. That can reduce recurring false positives and align review priorities with local conventions, but it does not independently establish construct validity or predictive validity. A change in an engineer's behavior is not a quality label. It can reflect attention, urgency, convention, or avoidance as well as a correct diagnosis.
This distinction matters because the model can flag poor human-written code and miss polished AI-written code. It is not a detector of provenance. It is a deterministic, inspectable hypothesis about structural review risk.
So we operationalized a structural-risk hypothesis
Our answer was not to define slop as a binary label. It was to operationalize four candidate observables for structural review risk. Each is inspectable, deterministic, and contestable; none is a validated measure of software quality by itself.
- Logic Density Ratio (LDR):
logic_lines / total_lines. It is a proxy for files whose apparent size overstates executable substance. It can be wrong for schemas, adapters, declarative configuration, type contracts, and test fixtures, so role classification and explicit skips matter.
- Inflation: effective jargon density adjusted by local complexity. It is a proxy for rhetorical or documentary claims that outpace the visible implementation. It is especially vulnerable to domain vocabulary, which is why a jargon score is a review prompt rather than a verdict.
- Dependency Usage Ratio (DDC):
used_runtime_imports / runtime_imports. It measures whether declared runtime imports participate in executable code, excluding annotation-only imports, TYPE_CHECKING branches, and intentional re-exports. Package existence and registry validity are evaluated separately through phantom-import rules; DDC alone cannot detect an invented package that is imported and used.
- Critical-pattern burden:
purity = exp(-0.5 * n_critical_patterns). This gives critical findings a continuous effect inside the composite. The current implementation also applies a severity-weighted additive pattern_penalty over the same issue list, including critical findings. That is explicit severity amplification, not an independent fifth measurement, and it is a design choice that requires empirical evaluation rather than rhetorical justification.
The score therefore has four structural proxies plus an additive policy term. The output retains a deficit_breakdown, fired rules, and a suppression ledger so that a reviewer can inspect the reason for a score instead of treating it as an oracle.
The final composite is not an arithmetic mean. It is a weighted geometric mean because a weak dimension should retain more influence than it would under an arithmetic average. This is the production method in src/slop_detector/core.py:
weights = self.config.get_weights()
w_ldr = weights.get("ldr", 0.40)
w_inf = weights.get("inflation", 0.30)
w_ddc = weights.get("ddc", 0.20)
w_pur = weights.get("purity", 0.10)
gqg = exp((
w_ldr * log(max(1e-4, ldr.ldr_score))
+ w_inf * log(max(1e-4, 1.0 - inflation_normalized))
+ w_ddc * log(max(1e-4, ddc.usage_ratio))
+ w_pur * log(max(1e-4, purity))
) / (w_ldr + w_inf + w_ddc + w_pur))
base_deficit_score = 100 * (1 - gqg)
deficit_score = min(base_deficit_score + pattern_penalty, 100.0)
The floor at 1e-4 is not cosmetic. It keeps the log transform defined while retaining strong sensitivity to weak dimensions. A weighted geometric mean is more weak-link-sensitive than an arithmetic average, while still allowing limited compensation across signals. The initial 0.40 / 0.30 / 0.20 / 0.10 weights are design defaults, not externally estimated parameters.
What the model does in practice
The detector runs a bounded operational loop: scan a file or repository, decompose the score into LDR, inflation, DDC, and pattern pressure, inspect the dominant signal, patch or narrow the finding, then rescan. The difference between consecutive runs is an operational signal for review, not evidence that quality improved.
Project-scoped history records repeated scans. A changed file whose score falls past the configured threshold is recorded in code as an improvement label; in this article it is an improvement candidate. A stable flagged file becomes an fp_candidate. Neither is an independent quality label.
A structured report can direct attention to imports, empty paths, role boundaries, critical patterns, and explicit suppressions that ordinary generation may miss. But a finding does not become useful merely because the detector emitted it.
It becomes useful after a reviewer accepts, rejects, or narrows it. An accepted finding may become a patch or test. A rejected finding may become a role rule, suppression rationale, or explicit repository exception. In the current implementation, those decisions can be preserved through project history, configuration, and suppression records. They are not yet guaranteed to be retrieved automatically into every future agent context.
A finding became useful only after it was adjudicated, recorded, and made available as a repository boundary for the next review cycle.
Adjacent work such as Reflexion 10, Self-Refine 11, and RepoCoder 12 suggests that explicit feedback and retrievable local context can influence later agent behavior without weight updates. That supports an evaluation hypothesis, not a result already demonstrated by this detector.
The appropriate test would compare a baseline coding workflow with a report-and-adjudication workflow, then measure recurring boundary violations, human-accepted changes, review burden, and false-positive handling. A lower detector score alone would not count as evidence.
What first-party dogfooding did, and did not, show
We exercised the detector across internal codebases spanning deterministic governance workflows, mathematical computing, BioAI pipelines, and finance-facing backend work. The useful observation was that the same static signal can mean different things in different roles: orchestration can be sparse but load-bearing; scientific notation can be dense but necessary; defensive validation can be repetitive but justified.
That is why the implementation has role rules, exclusions, domain anchors, and a suppression ledger. These mechanisms make the boundary of a finding explicit: not simply “this pattern is bad,” but “this pattern is suspicious unless this file role, domain convention, or documented exception applies.”
It is not an external experiment. The codebases share one project ecosystem, with no blinded labels, independent reviewers, published precision/recall, or baseline comparison. The experience shows that false-positive boundaries can be revised in varied internal work; it does not show that the score measures a unified latent condition, predicts defects, or generalizes.
The next useful artifact is a public case record: detector warning, human acceptance criterion, reason the proxy failed, boundary change, rescan result, trade-off, and residual uncertainty. Until such evidence exists, this remains an operational account rather than a validation study.
The calibration algorithm
The weighted geometric mean is only the scoring surface. The operational-adjustment mechanism in src/slop_detector/ml/self_calibrator.py is repository-scoped and history-driven.
- Every recorded run carries a
project_id, so local history is not mixed by default.
- The automatic milestone is based on files scanned more than once, not raw scan count, because only repeated files can create a before-and-after record.
- The engine requires at least five
improvement events and five fp_candidate events before searching for a new profile.
- Search is constrained around the domain anchor, limiting arbitrary drift from the current configuration.
- The code field is named
confidence_gap, but it is not a statistical confidence interval. It is the objective separation between the best and second-best weight candidates; if the separation is too small, the engine returns insufficient_data.
- Git metadata filters obvious measurement noise on both labels. Neither is an independent ground-truth signal: a score drop can be gamed, and an unchanged file can just as easily reflect a lower-priority task as a genuine false positive.
- Those claims are also enforced in code. The calibration path first refuses thin evidence, then compares candidate profiles by error and a tie-break, and finally refuses to choose when the apparent winner is not meaningfully separated:
improvements = [e for e in events if e.label == "improvement"]
fp_candidates = [e for e in events if e.label == "fp_candidate"]
if len(improvements) < min_imp or len(fp_candidates) < min_fp:
return CalibrationResult(status="insufficient_data")
candidates.sort(key=lambda c: (c.combined_score, c.tiebreak_score))
winner = candidates[0]
if len(candidates) >= 2:
runner_up = candidates[1]
primary_gap = runner_up.combined_score - winner.combined_score
if abs(primary_gap) < 0.0001:
confidence_gap = (runner_up.tiebreak_score - winner.tiebreak_score) / max(
1.0, abs(winner.tiebreak_score)
)
else:
confidence_gap = primary_gap
else:
confidence_gap = 1.0
if confidence_gap < CONFIDENCE_GAP:
return CalibrationResult(status="insufficient_data")
The production path additionally rejects a score drop inside the same Git commit as measurement noise, and rejects a stable file whose surrounding commit changed as ambiguous. These are useful safeguards for operational adjustment. They do not convert behavioral history into an external quality label.
The practical result is narrower than a self-improving quality model. A team can start with a default profile, run the detector against real repository changes, and use recurring local behavior to review whether a signal is over- or under-sensitive.
That is the part that turns a formula into an inspectable operational artifact rather than a one-size-fits-all lint rule. It does not prove the formula is valid. It makes the formula, its thresholds, its suppressions, and its revisions available for challenge.
The implementation and its limits are open source
The logic described above, the proxy definitions, the weighted geometric composite, the disclosed additive severity term, and the operational-adjustment path are released as open source in the AI-SLOP Detector repository. The validation boundary is released with them. Open source does not validate a metric; it makes the implementation, its assumptions, and its failure modes available for inspection and independent testing.
Why operationalize a concern at all?
A soft concern in prose is difficult to compare, challenge, or reproduce. An explicit heuristic gives a reviewer observable inputs, a decision rule, an evidence trail, and a specific term to dispute. That is useful for triage, not a substitute for tests, formal verification, change control, dual approval, or human review.
The engineering advantage is auditability, not automatic truth. A scalar score can prioritize inspection; a deterministic gate can prohibit an unsafe transition; a human may remain the final authority. An explicit heuristic can still be consistently wrong, but its assumptions are visible enough to test, revise, or reject.
Conclusion
Capability is rising, but benchmark scores do not establish that a model can find unseen faults or preserve architectural quality. Review remains the scarce resource, and plausible code can consume it without earning trust.
This project does not claim to have mathematically solved AI slop. It makes a narrower claim: structural review concerns can be expressed as deterministic, auditable proxy signals. In this implementation, LDR, inflation, DDC, and critical-pattern burden are combined with a weak-link-sensitive geometric composite and an explicitly disclosed additive severity term that may overlap with the critical-pattern signal. Repository-local history can adjust review sensitivity, while explicit data floors, Git noise filters, and objective separation between candidate profiles can refuse weak recommendations.
That is operational calibration, not independent validation. The tool has not yet shown that its score measures a unified latent condition, predicts defects or maintenance cost, outperforms simpler baselines, or generalizes beyond its development environment. Those are empirical questions requiring blinded labels, external reviewers, out-of-sample repositories, and published failures.
The contribution for now is an inspectable engineering hypothesis: a way to make a vague concern explicit enough to compute, audit, dispute, suppress with a record, and test against stronger evidence. A number is not a quality bar by itself. It becomes useful only when its assumptions remain visible and challengeable.
References
- OpenAI, "Why SWE-bench Verified No Longer Measures Frontier Coding Capabilities," February 23, 2026
- OpenAI, "Separating Signal From Noise in Coding Evaluations," July 2026 (SWE-bench Pro audit and retraction)
- "GBQA: A Game Benchmark for Evaluating LLMs as Quality Assurance Engineers," arXiv:2604.02648
- Farrag, "The Productivity-Reliability Paradox: Specification-Driven Governance for AI-Augmented Software Development," arXiv:2605.01160 (May 2026)
- Imai, S., "Is GitHub Copilot a Substitute for Human Pair-Programming? An Empirical Study," ICSE '22 Companion Proceedings, 319–321 (2022)
- Churilov, "The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on the 2026 Frontier-Model Cohort," arXiv:2605.17062
- Kommers, Duede, Gordon, Holtzman, McNulty, Stewart, Thomas, So, and Long, "Why Slop Matters," ACM AI Letters, Vol. 1 (2026), arXiv:2601.06060
- "An Endless Stream of AI Slop: The Growing Burden of AI-Assisted Software Development," arXiv:2603.27249 (March 2026)
- Zhou, Saghi, Sabouri, Pandita, McGuire, and Chattopadhyay, "Cognitive Biases in LLM-Assisted Software Development," arXiv:2601.08045 (2026)
- Shinn et al., "Reflexion: Language Agents with Verbal Reinforcement Learning," arXiv:2303.11366
- Madaan et al., "Self-Refine: Iterative Refinement with Self-Feedback," arXiv:2303.17651
- Zhang et al., "RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation," arXiv:2303.12570
- Zechner and Ronacher, interview on "vibe slop," The Wall Street Journal, May 23, 2026, as reported by InfoWorld
- Ronacher, "Pi: The Minimal Agent Within OpenClaw," lucumr.pocoo.org, January 31, 2026
- Harding and Kloster, "Coding on Copilot," GitClear, 2024/2025 longitudinal analysis
