Originally published at bryanhamilton.info
Search engine optimization has a new sibling. If Google decides whether to rank you, that's SEO. If a language model decides whether to recommend, cite, or summarize you - that's GEO: generative engine optimization. Same discipline as ever (be legible to machines), new examiner.
Here's how I know it matters: an AI-visibility audit scored my site 24/100. "Needs improvement - AI could not reach your site."
Which was baffling, because my robots.txt explicitly welcomed the AI crawlers by name. GPTBot, ClaudeBot, OAI-SearchBot, the whole guest list.
The audit that lied (and the toggle that didn't)
Permissive robots.txt, unreachable site. That contradiction is the tell: something between the bot and Apache was answering for me. So I ran the cheapest diagnostic in the world - a user-agent matrix:
for ua in GPTBot ClaudeBot OAI-SearchBot Claude-Web PerplexityBot \
Googlebot Bingbot "Mozilla/5.0"; do
printf '%-16s %s\n' "$ua" "$(curl -s -o /dev/null -w '%{http_code}' \
-A "$ua" https://example.com/)"
done
One signature mattered: a 25-byte, text/plain 403. My PHP never ran. Apache never logged it. That's a WAF-shaped hole - and it was Cloudflare's "Block AI Scrapy" toggle, doing exactly what it says on the tin.
The fix was one toggle. The diagnosis was the matrix. After flipping it, all 39 user-agents in my full sweep returned 200.
Re-run your audits, always
Here's the part that saves you from shipping a ghost: the audit still said 24/100 after the fix. Verdicts are snapshots. The tool had cached its crawl, the same way every cache in this stack does. Re-ran the scan - clean. An infra fix is not done when it's deployed; it's done when a fresh measurement says so.
The file layer: llms.txt and ai.txt
llms.txt is a curated map of your site for language models - markdown, human-readable, with a short description of what the site is and links to the good stuff. ai.txt is a variant with the same job. My setup on both domains:
- llms.txt at the root: what the site is, the evergreen URLs, the writing hub
- ai.txt as a byte-identical copy of llms.txt (verified with md5sum, not by trusting my own copy-paste)
One wrinkle: the parser used by at least one auditor choked on typographic characters, so the file got a pure-ASCII rewrite. Em-dashes are not worth a failed ingest.
IndexNow: tell the indexes now, not eventually
IndexNow is the ping protocol Bing and friends accept: you signal "these URLs changed" the moment they change, instead of waiting for a crawl schedule. Implementation is two files and one habit - the key file proving you control the domain, and a batched ping on every publish and every removal. A dead URL in the index is as bad as a missing one.
curl -s -X POST https://api.indexnow.org/indexnow \
-H 'Content-Type: application/json' -d @batch.json
(The habit bit me this week, actually: my sitemap generator silently dropped two pages on a regen, and the IndexNow ping is what forced me to notice. The plumbing pays for itself.)
The 429 red herring
One bot saw an HTTP 429 during testing. My limiter-phobia said "I'm being blocked!" - no. The site runs a per-IP rate limiter, and a burst of test requests looks like a burst of test requests. It cleared in under a minute. Diagnose transients before you patch around them; a fix for a transient is a bug with extra steps.
Entity hygiene: canonicals and sameAs
LLMs build entity pictures the same way search engines do: who is this, what else do they appear as. The boring fundamentals carry the weight:
- Every syndicated article's canonical points at the self-hosted original. (dev.to authors: the
canonical_url front-matter overrides the API's canonical parameter. Ask me how I know.)
sameAs in JSON-LD mirrors every real profile - exactly once, no duplicates, no aspirational handles. This one bit me again just this week: a social URL I'd listed turned out to be somebody else's account. Verify every URL like a hostile auditor wrote it.
The checklist
- UA matrix over robots.txt - verify reachability, not intention
- robots.txt welcomes the AI bots by name (or deliberately doesn't)
- llms.txt + ai.txt, ASCII-clean, byte-verified
- IndexNow key live, ping on publish AND remove
- Canonicals point home on every syndication
- Re-run audits after infra changes - never trust a cached verdict
None of this is glamorous. All of it is the difference between "the AI never heard of you" and "the AI quotes your production notes back to a hiring manager." The first audit said 24/100. The fresh one says the bots are through the door - because I checked what they actually see, not what my robots.txt wishes they saw.
Question for the comments: have you ever run a user-agent matrix against your own site? Half the devs I ask have never checked whether GPTBot can actually reach them - they just assume their robots.txt is the truth. Run the loop above and tell me what you find. I'm betting at least one of you has a WAF toggle silently eating your AI traffic.