How AI Companies Turned Your Thinking Into Their Training Data
Here's a thesis worth sitting with, even if you end up rejecting it: the most valuable resource an AI company can extract from you isn't your data. It's your reasoning. And the mechanism for extracting it is already live, hiding in plain sight, dressed up as a feature.
We've been here before. CAPTCHA — "completely automated public Turing test to tell computers and humans apart" — was sold to the internet as a security checkpoint. Click the traffic lights. Type the squiggly letters. Prove you're human. What nobody advertised was that those clicks were never just security theater. Google's reCAPTCHA, most famously, repurposed millions of users into an unpaid labelling workforce, training image recognition systems and digitizing books and street signs at planetary scale. You weren't proving your humanity. You were teaching a machine to see. The Turing test was the bait; the dataset was the catch.
That playbook didn't disappear. It evolved. And the new version doesn't target your clicks — it targets your thoughts. Call it, if you like puns as much as you dislike being farmed for data, a Completely Automated Public Turing test to Capture your Human Argumentation. The acronym still spells CAPTCHA. The product has just been quietly upgraded.
The New Mechanism
Consider how often a modern AI chat stalls mid-conversation, asking you to clarify, re-explain, or restate your reasoning before it'll proceed. The official story is that it's being careful, avoiding hallucination, making sure it "understands." But look at the pattern more closely. The moments these systems balk most reliably are exactly the moments where your logic chain has a gap in it — where you've jumped a step, assumed a conclusion, or left a thread of reasoning incomplete.
What happens next is the part worth interrogating. You don't just get a refusal. You get a prompt — explain your reasoning, walk through your logic, restate your goal in clearer terms. And you comply, because that's what a good-faith collaborator does. You fill in the gap/ the option modal; you externalize the part of your thinking that used to live only in your head.
That externalized reasoning doesn't evaporate. It's logged, structured, and — by extension of every public training disclosure these companies have already made about using user interactions to improve their models — fed back into the system. You didn't just get unstuck. You taught the model how a human bridges a logical gap. Multiply that by tens of millions of conversations, across every domain of human expertise, and you have something no scraped dataset could ever buy: a live, continuously updating map of how people actually think when they're trying to solve real problems they care about.
This is CAPTCHA's core trick, upgraded. CAPTCHA harvested perception — what does a crosswalk look like. CAPTCHA 2.0 harvests cognition — what does a sound argument look like, what does a person do when their logic breaks, how does a human course-correct mid-thought. It's a vastly more valuable dataset, and it's being collected under the banner of "helpfulness." Old CAPTCHA asked you to prove you weren't a robot. New CAPTCHA asks you to prove you can think like one wants to — and then keeps the receipts.
Why Stalling Is the Feature, Not the Bug
Skeptics will say this is paranoid — that models pause on broken logic because broken logic genuinely produces bad outputs, and asking for clarification is just good design. Maybe. But good design and good data extraction aren't mutually exclusive; they're frequently the same mechanism wearing different labels. A company doesn't need a secret memo instructing engineers to "harvest reasoning chains" for the effect to be real. It only needs an incentive structure where models that prompt more human elaboration also generate more valuable training signal — and where that correlation never quite gets resolved in the user's favour.
The incentive is structural, not conspiratorial. That's what makes it durable.
What This Costs You
The price of CAPTCHA 1.0 was a few seconds of your attention, repeated billions of times. The price of CAPTCHA 2.0 is steeper, because what's being extracted isn't a perceptual judgment you'd have made anyway — it's the specific architecture of your problem-solving. Your hesitations. Your false starts. The exact moment you noticed your own assumption was shaky and corrected it. That's not noise to a model trainer. That's signal of the highest order, because it's the part of human cognition that's hardest to fake and most expensive to simulate synthetically.
And you're not being paid for it. You're not even being told it's happening, because there's no clean line between "the model needed clarification to help you" and "the model's clarification request doubled as a data-collection event." Both can be true at once, and the company has no incentive to help you tell them apart. Select all squares containing a coherent thought. If none apply, click "verify" and try thinking again.
The Prediction
Here's where I'll stake a claim: this won't stay confined to one or two labs. It will become standard practice across the AI industry, because the economics are too good to ignore. Every major AI company will, in some form, build products that lean on user-supplied reasoning to keep their models sharp — and every one of them will have a plausible, even sincere, justification for why it's simply good UX.
And here's the second half of the prediction, the part that should actually worry these companies: they will also actively work to slow the rise of local, self-hosted LLMs — models that run on your own hardware, train on your own data, and report to no one. Local models break the extraction loop entirely. There's no conversation to log, no reasoning to harvest, no behavioural signal to sell upstream. Expect friction: licensing restrictions, compute gatekeeping, "safety" framing applied selectively to open-weight models, pricing structures that make proprietary access cheaper than self-hosting at small scale. Expect it to work — for a while. And expect it to ultimately fail, because the same economic pressure that makes harvesting attractive to incumbents makes independence attractive to everyone else.
A Step in the Right Direction
Not everyone building frontier models is racing to lock you deeper into the extraction loop. Subquadratic is worth watching here — their SubQ architecture trades the standard quadratic-attention transformer for a sub-quadratic, sparse-attention design that claims roughly 64.5x less compute than dense attention at long context lengths. Whatever you make of the specific benchmarks, the direction matters: smaller compute footprint, lower production cost, less infrastructure pressure to monetize every interaction just to keep the lights on. An industry where running a capable model doesn't require planet-scale compute is an industry with more room for the local, sovereign alternative this piece is arguing for. It's not the whole answer — efficiency doesn't automatically mean ownership — but cheaper, leaner architecture is a precondition for it.
I'm tracking this space closely myself, working through a spectral equivalence approach to LLM architecture — treating attention and efficient long-context mechanisms through the lens of spectral methods rather than brute-force quadratic comparison. It's early, it's mine, and I'll be posting updates on this blog as the work develops. Consider this the first marker.
Internet 2.0
If CAPTCHA 1.0 quietly built the datasets that power today's vision and language models, CAPTCHA 2.0 is quietly building the dataset that powers tomorrow's reasoning models — using you as the unpaid annotator of your own mind.
But extraction loops eventually meet their limit, and I think we're closer to that limit than the incumbents would like. As local compute gets cheaper and open models get sharper, the calculus shifts. Why train someone else's proprietary system with your best thinking when you could be training your own?
That's the real endpoint here — not a tweak to AI products, but a structural shift in who the internet serves. Call it Internet 2.0: a web of individuals running personal LLMs, models that learn from them and for them, not models that learn from them and sell the result upstream. Less a platform, more a population of sovereign minds, each running its own.
A renaissance, this time, that you actually own.
