You did the responsible thing with your robots.txt. You let the search bots in so they can cite you. You blocked the training bots so no model eats your content and paraphrases you to strangers without a link. You confirmed the site was crawlable and indexed. Textbook.
Then someone asks ChatGPT "what is a good tool for X," and you are not in the answer. Not ranked low. Absent. The work you did to be found is the exact thing hiding you.
Here is why, and why the fix is not the switch you think.
Many answers never search at all
Your name reaches a ChatGPT answer one of two ways. Either the model answers from memory, out of what it absorbed during training, or it runs a search and comes back with links.
The catch is that search is not guaranteed to run. On the free tier, where many casual users are, search is not enabled by default, and most people never turn it on. Even when it is available, the model decides per question whether it is worth the trouble. For news and prices it looks. For "what should I use," it often just answers from what it already knows.
So a lot of the questions that matter most to you get answered from memory. And your own site only reaches that memory if you let the training crawler in. Block it, and as far as your own pages are concerned, the model has no idea you exist the moment it answers without searching.
That is where the real decision sits. Not "how do I rank." To train, or not to train.
To train: what it costs you
Open the training crawler and you get into memory, so you can show up even when the model never searches. Here is the bill for that:
- You go stale. Foundation models retrain a few times a year. Your pricing, features, and positioning freeze at whatever the model last ingested, and stay wrong until the next retrain.
- No link. Memory answers do not cite. The model recites you without sending anyone to your site.
- No traffic, no attribution. You become a fact in a sentence, not a destination. People act on what the model says and never learn you were the source.
- Your content is in the dataset for good, used however the model sees fit, with no way to pull it back.
- And it still is not guaranteed. Being crawled for training does not mean the model will surface you. You paid all of the above and may still not show up.
Not to train: what it costs you
Keep training blocked and you protect your content, but you accept the other half of the trap:
- You are not in memory. Every answer the model gives without searching is an answer you cannot appear in.
- You are betting everything on search firing. And on the free tier, where most casual users live, search is not on by default and most people never turn it on. They ask, the model answers from memory, and you were never in the room.
One path makes you outdated. The other makes you invisible. That is why the title is a real question and not a rhetorical one.
First rule out the boring cause
Say you take the search side anyway: block training, allow the search crawler, hope to be cited when the model looks. Even then it can search and skip you, sometimes for a dull technical reason. Modern sites render in the browser with JavaScript, and crawlers often receive an empty shell instead of your content, so the model finds nothing to cite.
So before you conclude anything, settle one question: can the search crawler actually read your page? Fetch your site the way the crawler sees it, not the way your browser paints it. Swap in your domain and a phrase from your homepage:
curl -s -A "OAI-SearchBot" https://yoursite.com | findstr /i "your headline text"
- Nothing comes back. You have a rendering problem, not a robots.txt problem. The crawler is getting an empty shell, and no directive fixes it until the page serves its content in plain HTML. Fix that first.
- Your text comes back. Good. Access is not your issue, and that is the important finding, because it means you can stop touching config entirely. Every crawler setting is fine. The reason you are invisible is not technical.
If the crawler sees you and you are still invisible
Then stop tuning settings. You are simply not the answer yet. Nobody cites you, few sites mention you, and the model has no reason to pick you over the names it already knows. You are not losing on configuration. You are losing because you have not earned the position.
Which points at the only real fix, and it is not in your robots.txt.
The actual answer: build the site, not the config
Stop tuning crawler directives and grow the thing itself. Get mentioned in roundups, review sites, Reddit threads, the docs and articles other people write. Get indexed in more places by more sources.
Do that and both doors open at once, without the trade-off:
- You land in the model's memory through their pages, not your own. Those articles and threads get crawled into the training data themselves, so when enough trusted sources describe you, the model absorbs you along with them. You end up in its built-in knowledge without ever opening your own site to the training crawler, and it can name you even when it does not search.
- You get cited with real traffic when search does run, because those are the pages it retrieves and links.
- You stay fresh, because those sources keep getting re-crawled.
That is the version where ChatGPT recommends you even on a cold, no-search answer. Not because you flipped a switch, but because the web already agrees you are the answer.
To train or not to train is a toggle with no good side. Reputation is the setting that actually decides it, and it is the one you cannot type into a file. Everything else is just choosing how you would like to be invisible.
Nataliia Poladova, AI Product Builder and Automation Specialist.
