Originally published on angeo.dev.
This is a method write-up. If you want the findings, they are in the original. What follows is how the study was built, in the order you would build it, so you can run one on your own frame.
1. Decide the question so it cannot drift
The question was: do businesses AI assistants name repeatedly differ, on observable technical signals, from businesses the same assistants name once?
Write down what the question does not cover, in the plan, before anyone can be tempted. Here: every business was named at least once, so nothing in the study estimates the chance of being named at all.
2. Seal the analysis plan
Before any data:
- the expected effect size (no signal separating groups by more than 15 points)
- the refutation condition (any signal differing by 20+ points with the named group higher)
- group definitions and thresholds
- the exclusion lists
Hash the document, send the hash to a second party, keep the file. Amendments get their own hashes and must be sent before the data they concern exists.
shasum -a 256 analysis-plan.md
# 9b4ccf12629e… analysis-plan.md
The point is not ceremony. It is that in three weeks, when a number comes out awkward, you cannot silently redefine the group.
3. Split the instrument from the analysis
The answers came from a research partner who ran the engine queries; the scanning was mine. Neither side saw the other's inputs until my plan was sealed.
If you cannot get a second party, the next best thing is to build the corpus and freeze it before you write a single check.
4. Build the frame, and publish the losses
Roster rows 669
no resolvable domain -186
resolved to a different company -3
marketplaces and listing surfaces -12
duplicate rows collapsed -10
Unique domains analysed 458
scanned successfully 455
Two rules worth stealing:
- The marketplace exclusion list is fixed before any signal is read. 14 hosts, written down, not curated once you can see who is winning.
- Every row is accounted for. If your table does not add to the universe, a reader cannot tell what you dropped.
5. Define groups on the pre-registered thresholds only
- Head excluded first: anything appearing in 53+ of 264 answers.
- Cases: 3+ mentions across both runs and present in both.
- Controls: exactly one mention across both runs.
Businesses named exactly twice fall between the two definitions. They stay excluded. Moving a threshold to collect them is choosing a group after seeing the answers.
6. Scanner rules that keep the study defensible
The scanner is a plain HTTP client with a short list of checks. What matters is its manners:
- identifies itself in every request, with a URL to a public crawler policy
- obeys
robots.txt, including Crawl-delay
- at least two seconds between requests to the same host
- keeps no page content — presence and shape only
- can be blocked in two lines
That last rule has a consequence you have to plan for. Because no content is stored, any content-level finding needs a second, explicit pass with its own justification — which is exactly how the template finding below happened.
7. Pin the URL for anything you will measure twice
In the panel study that preceded this one, 23 of 196 stores — 12% — had a different product page sampled on the second visit, because the scanner discovers product pages by following a candidate link from the homepage.
Compared naively, a store that swapped a bicycle for a paperback looks like a store that lost its JSON-LD. Nothing changed. If you re-measure, store the tested URL and compare only where it matched.
8. Run the test, then interrogate the surprise
Two-sided Fisher exact on each 2×2 table. The primary result was null — no signal met the refutation condition.
One signal did move, in the wrong direction: llms.txt was 15 to 17 points more common among the once-named group.
Instead of shipping that as a finding, I fetched the 159 flagged files and read the first line of each. 113 matched one generated template, differing only in the brand name. Adoption measured as 32% dropped to 9% once the template was removed — against 11% on a separately measured frame of 762 Magento stores.
The lesson generalises: when a rate differs sharply between two populations, check whether you are measuring decisions or measuring one piece of software.
9. Publish the failure condition when it fires
A secondary pre-specified cut did produce a 22.5-point difference in the refuting direction — on a case group of four stores. Statistically meaningless. Reported anyway, because a refutation condition you quietly drop when it fires on n=4 was never a refutation condition.
10. Ship the artefacts
Sealed plan, amendments, hashes, exclusion lists, per-business rows, scanner. All published alongside the article.
Full study with every table and p-value: AI assistants named 458 shops
Crawler policy: angeo.dev/magento-ai-visibility-statistics
Have you run a measurement like this and had the surprise turn out to be an artefact of your own tooling? I would like to hear which one — mine was the template.
