Most robots.txt files in production were written years ago, for one crawler, with one goal: let Google in, keep it out of the admin panel.
That file is still doing its job. But it's now also answering a question nobody asked it: whether ChatGPT, Claude, Perplexity, and Gemini can use your content in the answers they generate.
For a lot of sites the answer is accidentally "no."
The part most people get wrong
The instinct is to treat "AI bots" as one category and make one decision about them. That's the mistake, because they are not one category — they're at least three, doing genuinely different things.
Training crawlers collect content in bulk to train foundation models. They're not urgent, they crawl broadly, and blocking them means your content isn't in the next training run.
Search crawlers index continuously so an assistant can cite live results when someone asks a question right now. Blocking these removes you from generated answers.
User-triggered agents fetch a specific URL because a person asked the assistant to look at it. Blocking these means assistants fail to open your pages on request.
These use separate user-agent strings and are independently controllable. That's the whole point: you can allow the ones that make you citable while disallowing the ones that feed training, if that's your preference.
A blanket Disallow for everything AI-shaped collapses that distinction and usually costs you the thing you actually wanted.
Why "block everything" is a worse default than it sounds
There's a reasonable instinct behind blocking AI crawlers — you wrote the content, why should a model consume it for free.
Two things complicate that.
First, if your content has commercial intent behind it — you're selling something, or you want to be found — blocking search crawlers removes you from an increasingly common discovery path. And it's hard to recover from, because you don't get retroactive visibility once you unblock.
Second, blocking training crawlers doesn't undo the past. Content published earlier may already sit in existing training corpora. The block affects future runs, not history.
None of which means "allow everything" is automatically right. It means the decision deserves more than one line.
Three things that are true and inconvenient
robots.txt is a convention, not enforcement. It works because well-behaved crawlers choose to read it. Some crawlers have been documented ignoring it. If you need a hard guarantee, that lives at the server or WAF layer — a robots.txt rule an agent ignores achieves exactly nothing.
User-agent strings are self-reported. Anything can claim to be anything. If you're making serving decisions based on crawler identity, verify against published IP ranges where the vendor provides them, or use reverse DNS confirmed by a forward lookup. Trusting the header alone is trusting a claim.
User-directed fetches are a gray area. When someone asks an assistant to summarise a specific URL, some providers treat that as a user action rather than crawling — so robots.txt may not apply the way you'd expect. This is unsettled, and reasonable people disagree about whether it should be.
The redirect trap
This one bites people who did everything else right.
Search-time agents have low tolerance for redirect chains. An extra hop can be enough for a page not to make it into a generated answer.
It's sharpest with llms.txt — the AI content-map file. If you serve it via redirect rather than directly at the root, some crawlers retry the root on the next pass instead of following the redirect. The file exists, your CMS reports it as configured, and it's functionally inoperative.
If you publish one, serve it at the root. Directly. No redirect, no rewrite chain.
What to keep blocked regardless
Whatever your AI policy, some paths have no discovery value and shouldn't be fetched by anything:
- admin and login paths
- cart, checkout, and account pages
- internal search result pages
- anything requiring authentication
This is unchanged from classic SEO practice. Worth restating because "we're opening up for AI crawlers" sometimes turns into opening up more than intended.
Review cadence
Set a quarterly reminder. The bot landscape genuinely moves: agents get introduced, split into separate crawlers as vendors separate training from search, and deprecated.
A robots.txt copied from a 2023 blog post will not have rules for agents that didn't exist in 2023 — and some of those agents are the ones that decide whether you appear in AI answers today. That's the most common failure mode I run into: not a deliberate block, just a file that stopped keeping up.
Get exact current user-agent names from vendor documentation rather than from articles like this one, including this one. Any list I write here has a shelf life measured in months.
If you run an ecommerce store
Everything above applies, and there's an additional layer.
Crawler access determines whether AI systems can reach your pages. It doesn't determine whether they can use your catalogue. For product data specifically, you also need complete structured data — a Product entity with a valid offers.availability, rendered server-side — and, for shopping surfaces, a product feed. AI shopping generally works on a push model: the merchant submits a structured feed rather than waiting to be crawled.
That means a store can have perfect crawler access and still not appear in product recommendations, which is a confusing failure to debug if you assume robots.txt was the whole job.
For Magento specifically I've written the full signal set up at angeo.dev, including the CLI tooling to check it — free and MIT-licensed, if that's useful. But the general principle holds on any platform: access is necessary, not sufficient.
Summary: treat AI crawlers as three families, not one. Allow the search crawlers if you want to be citable. Decide training separately. Verify identity rather than trusting headers. Don't redirect llms.txt. Review quarterly, because this is moving faster than your robots.txt is.
