The robots.txt rule that looks like it blocks AI crawlers and does not
Here is a robots.txt that appears to keep OpenAI's crawler out of two sensitive paths:
User-agent: *
Disallow: /checkout/
Disallow: /customer/
User-agent: GPTBot
Crawl-delay: 10
It keeps GPTBot out of nothing.
Under RFC 9309, a crawler that matches a named user-agent group uses only that group. It does not inherit rules from User-agent: *. GPTBot reads its own group, finds one Crawl-delay and no Disallow, and crawls everything — including the two paths above.
The moment you name a bot in order to be more careful with it, you switch off every rule you had written for it. And the failure is silent: the file reads as though it says something.
How to check yours
Fetch your own file and ask one question per named group: does this group repeat every Disallow you meant to apply?
curl -s https://example.com/robots.txt
If you have a User-agent: GPTBot block and its rules are shorter than your wildcard block, that difference is the hole. Either delete the named group so the bot falls back to the wildcard, or repeat the full rule set inside it. There is no third option — inheritance does not exist here.
Named AI crawlers worth checking for, if you are auditing: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Google-Extended.
Why I went looking
I spent a few weeks cataloguing every project built for one ecommerce platform to make stores legible to AI systems — 42 of them, for Magento. Grouped by what they do, the distribution was lopsided in a way I did not expect:
llms.txt discovery files — 11 projects- Specifications — 7
- MCP servers — 6
- Agentic checkout — 5
- Structured data — 4
- Crawler policy — 3
- Product feeds — 3
- Auditing — 3
Eleven implementations of a markdown file at your site root. Three for the layer where the bug above lives.
The easiest layer got built eleven times and the hard layers three times each. Crawler policy is not glamorous, it does not demo well, and it is where the actual failures are.
The order I would work in
- Measure. Bing Webmaster Tools reports real citation counts under its AI Performance view — the only free first-party source I know of that does.
- Fix crawler policy. Read the RFC first, then verify the rule.
- Structured data, then
llms.txt, in that order.
- Stop. Everything past that is early, and being early is a cost rather than a badge.
The list is CC0 if it is useful to you: awesome-magento-aeo. Disclosure — I maintain 11 of the 42 entries, which is why my projects are listed last within their sections rather than in alphabetical position.
Originally published at angeo.dev.
