The check was fine. That is the part worth sitting with.
I was auditing the structured data on our own site, the sameAs block that tells search engines which other profiles belong to the same company and the same person. It is a small thing that has to be exactly right, because a URL in there that points at nothing is not a neutral mistake. It is a broken claim about identity, published on every page.
One of the entries was a Crunchbase profile. Crunchbase returns 403 to anything that is not a browser, so curl could never settle it, and that entry had been sitting in the code for weeks carrying a comment that said, honestly enough, "unconfirmed, needs a look in a real browser".
So I looked in a real browser. It said Page not found.
Not a challenge screen. Not a login wall. An h1 reading Page not found, and body text about space cadets being unable to find the page. I had even written the check carefully: it tested the page for the usual bot-block giveaways, just a moment, access denied, are you a robot, and reported that none of them were present. A clean negative. So I removed the URL from the Person entity and moved on to write the commit message explaining why a dead link had to go.
The profile exists. It has a photo, a biography, an education entry, and a website field pointing back at our own site. Crunchbase serves logged-out visitors a page that says the profile is not there.
I had built a detector for lying, run it, got the all-clear, and been lied to anyway!
The shape of the problem
The familiar warning is that absence of evidence is not evidence of absence. That is true and it is not quite the thing. It suggests the danger is a check that comes back empty, and empty results at least look empty.
What actually happened is worse. The instrument returned a positive result about the wrong question. It could not see profiles, so it truthfully reported on the page it was given, which happened to be a page about not finding things. Nothing errored. Nothing timed out. Nothing was ambiguous enough to slow me down.
Once I noticed it, I went back through the week and found the same shape five more times.
The field that measures something adjacent
I wanted to know whether a personal GitHub profile had anything worth linking to. The API says:
"public_repos": 0
Zero. So there is nothing there, and the link should point at the organisation instead.
public_repos counts repositories the account owns. It says nothing about what the profile page renders. Fetched anonymously, that same profile shows six pinned repositories from the organisation and a contribution graph with 68 active days, because commits authored under a verified email are attributed to the account regardless of which repository they landed in.
The number was accurate. It was answering a question I had not asked.
The endpoint that disagrees with itself
Same tool, same API, same week. Auditing licences across the organisation:
gh repo list myorg --json name,licenseInfo
# → licenseInfo: null, for every repository
Five repositories, all reported as having no licence. That is a real problem if true, because an unlicensed public repository is all rights reserved by default, which is the opposite of what an open source project wants.
Then, per repository:
gh api repos/myorg/some-plugin --jq .license.spdx_id
# → "GPL-2.0"
Every one of them. LICENSE at the root, detected correctly, reported correctly. The list endpoint and the item endpoint give different answers about the same field, and the list endpoint is the convenient one, which is why it is the one you reach for when you are checking five things at once.
The channel that only exists for humans
I spent an afternoon convinced our analytics were not installed, because this returns nothing:
curl -s https://example.com | grep -c 'beacon.min.js'
# → 0
Two independent reasons, and each alone is enough. The provider only injects the snippet when it believes a real browser is asking, so a curl never sees it. And the page view is posted same origin, to a path on your own domain, so grepping the HTML for the analytics vendor's hostname finds nothing even when the page view is being recorded perfectly.
Two blind spots stacked on top of each other, and the observable result is a confident zero.
The write that goes nowhere
This one cost the most.
Our CMS holds a File System Access handle to the repository, granted once by the browser. That permission can lapse back to denied without the editor noticing. When it does, the editor still accepts your text, still greys out the Save button afterwards, and still shows the document as saved. Nothing reaches disk.
A fully reviewed Arabic translation, forty-eight changed lines, existed nowhere but in a tab.
A near-identical thing happened on a profile platform in the same week. I edited the alt text on an image, saved it, and the live page kept serving the old string through a no-cache, must-revalidate response and two cache-busted fetches, so staleness was not the explanation. That platform binds alt text to the uploaded image, not to an editable field. Editing the field changes nothing. Re-uploading the image is the only write that lands.
In both cases the interface reported success. The artefact disagreed.
The success that isn't
gh release create v1.0.0 --notes-from-tag --verify-tag
# → https://github.com/myorg/repo/releases/tag/v1.0.0
A URL came back. The release exists. Its title is null, while every other release in the organisation is titled with its tag name, so the one release created by following the documented command is the only non-conforming release in the set.
The command did not fail. It did something slightly different from what every previous release had done, and reported the same success either way.
And the error message I read as data
The smallest one, and my favourite, because there is nowhere to hide.
I fetched a plugin's source to count something in it, using the wrong filename. The API returned a JSON error body. My script counted characters in the response, found what it was looking for zero times, and reported the file as clean.
It was a perfectly accurate measurement of a 404 message.
Had I acted on it, I would have deleted a line from a test fixture whose entire purpose was to contain the character I was counting.
What actually catches these
Not care. I was being careful in every one of these cases. Being more careful is not a technique.
Run the check against something you know is there.
That is the whole method. Before you accept that a check found nothing, point it at a case where the answer must be positive and confirm it finds that. If the check cannot find the thing you know exists, it cannot tell you anything about the thing you are asking about.
I did do this correctly, twice, in the same week, which is how I know it works. Verifying two profiles on platforms that return 200 for everything, including missing pages, I checked a deliberately invented handle alongside the real one:
hashnode.com/@realhandle → <title>Jaafar Abazid (@realhandle) | Hashnode</title>
hashnode.com/@zzqq-not-a-real-user → <title>User not found | Hashnode</title>
The status codes were identical. The titles were not. The contrast is the signal, and it exists whether or not you think to look for it.
On Crunchbase I never ran the contrast, because a page that says Page not found felt like the end of the enquiry rather than the start of one.
The checklist I use now
- Before concluding absence, prove the instrument can detect presence. One known-positive control, every time.
- When an aggregate is convenient, spot-check one item against the per-item endpoint. List views and item views are different code paths and they drift.
- Read the field name literally.
public_repos counts owned repositories. It is not a synonym for "has anything on their profile".
- Verify browser-only behaviour in a browser, and verify logged-in behaviour logged in. A platform can serve three different truths to
curl, to a logged-out browser, and to you.
- After a write, re-read the artefact rather than the editor. The interface reports its intention. Only the artefact reports the outcome.
- Never count anything in a response you have not confirmed is the response you asked for. Check the status before you parse the body.
The part I keep coming back to
A check that fails tells you it failed. You retry it, you read the error, you fix the call. It is noisy and it is honest and it costs you five minutes.
A check that cannot see the thing you are asking about does not fail. It answers a neighbouring question, in the same format, with the same confidence, and hands you a result that fits neatly into whatever you were about to do next.
The Crunchbase URL is still in the structured data on every page of mantek.io. It survived because a screenshot arrived before the commit did, not because the process caught it.
That is not a comfortable thing to publish. It is the reason the control case is now the first thing I write, before the check itself.
