Sometime in the last two years you probably added a rule to robots.txt to keep AI crawlers out. A lot of us did. The intent was reasonable: don't train on my content without asking.
Go look at what you actually wrote. There is a good chance it reads something like this:
User-agent: *
Disallow: /
Or it is a wildcard rule inherited from a config written for Google in 2019, which predates every agent this is supposed to be about.
Either way, the result is not what you intended.
OpenAI runs several agents and they do different jobs
GPTBot — crawls for model trainingOAI-SearchBot — crawls to build the index behind ChatGPT searchChatGPT-User — fetches a page when a user's prompt triggers it
They are separate on purpose. Blocking all of them opts you out of training and removes you from ChatGPT's search index. The first was the goal. The second was collateral, and it means your content can no longer be cited when someone asks a question your site answers.
If what you wanted was "don't train on me, but do cite me":
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
That is a defensible position. So is blocking everything. So is allowing everything. The problem is that most robots.txt files I look at are not expressing any of those positions — they are expressing a misunderstanding.
The part that surprises people
Fixing this does not put your products in ChatGPT's feed-driven shopping results, because that surface is not crawled.
Product data for it reaches ChatGPT through a feed the merchant pushes to OpenAI. In March 2026 OpenAI moved away from Instant Checkout and put its weight behind discovery instead — get the catalog data right, let the purchase complete on the merchant's own site.
But there is a third route that most write-ups miss, and it is the one that makes the robots.txt question matter more, not less. ChatGPT also runs Shopping Research, an agentic mode that browses in real time. OpenAI says its results are organic and based on publicly available retail sites — reading product pages directly and citing sources. Merchants who want to be eligible go through a separate allowlist.
So there are three pipelines, and people routinely collapse them into one:
- crawlers → citations in conversational answers
- feeds → placement in feed-driven shopping results
- live browsing → inclusion in Shopping Research, which reads your actual pages
Blocking crawlers costs you the first and the third. Google is closer to the simple version: AI Mode draws on a Shopping Graph built mainly from Merchant Center feeds, with your site markup contributing rather than deciding.
Perplexity is the odd one out. PerplexityBot indexes your actual HTML, so page quality still translates into visibility there with no feed and no application. Worth knowing, though: Perplexity's own docs say Perplexity-User — the live fetch triggered by a prompt — generally does not honour robots.txt. Whether that is defensible is a separate argument, and Cloudflare and Perplexity have been having it publicly since August 2025.
Check your logs, not your config
grep -Eo "GPTBot|OAI-SearchBot|ChatGPT-User|PerplexityBot|Googlebot" \
/var/log/nginx/access.log | sort | uniq -c | sort -rn
This matters more than reading your own robots.txt, because firewall-level blocking is invisible there. Your rules can be perfectly permissive while Cloudflare or your host rejects the request before anything is served. The log is the only ground truth about what is actually reaching you.
Where I land, and where I might be wrong
I think the crawler distinction is worth getting right regardless of your position on AI training, because "I opted out of citations by accident" is nobody's actual position.
What I am less sure about: whether being cited by ChatGPT is worth much. Citation rarely converts to a click — the answer usually satisfies the user in place. There is a real argument that being indexed by an engine that answers on your behalf is a bad trade, and that the accidental block was the right call arrived at by the wrong route.
I do not think that argument is obviously wrong. I just think it should be a decision rather than a typo.
So: did you block AI crawlers deliberately, and did you check what you actually blocked?
I build open-source AEO modules for Magento, so I have a stake in people caring about this. Treat it as a claim to verify rather than accept — the log command above is the verification.
Longer version covering all three engines: angeo.dev
