Every codebase I've seen that touches the live web has the same test folder: empty, or holding one test marked skip with a comment that says "runs against prod, do not enable."
It is not laziness. It is a reasonable-sounding belief: you cannot unit-test against a website, because a website is not a fixture. It changes. It rate-limits. It serves different HTML to different clients. So testing gets declared impossible, and correctness is delegated to production — which is another way of saying your users will run your tests for you, one bad row at a time.
The belief is wrong in a specific way. You cannot test against the live web, true. But almost nothing in a scraper actually needs the live web, and the part that does can be reduced to one cheap, scheduled check.
Three layers, none of them optional
Layer 1: the parser, against fixtures. Record a real response once. Commit the raw HTML as a golden file, and next to it, the exact parsed output you expect. The test now runs in milliseconds, offline, deterministic. This layer catches the boring bugs: the optional field you forgot to handle, the currency symbol you never stripped, the second item in a list you always assumed was the first.
Layer 2: the parser, against its own failures. Almost nobody does this one, and it is the most valuable. Take your golden file and mutate it on purpose: rename the class your selector depends on, delete the price node, swap the unit string from kg to g. Then assert that your parser fails loudly — a validation failure, a rejected record, an exception — instead of quietly returning None and calling the record valid. A parser that has never been shown a page it cannot understand will not fail loudly in production either.
Layer 3: the fixture, against the site. Fixtures go stale, and that is not a flaw; it is the design. One scheduled job, weekly or nightly, fetches the live page once, runs the real parser, and diffs the output against your committed expectations. One request. If it matches, you do nothing. If it drifts, you now hold three things in the same commit: the new page, the old expectation, and the diff between them. That is the raw material of a fix, produced before any downstream table is touched. This job exists to keep your fixtures honest. It is not your test suite, and it must never run on every push.
What this replaces
The compromise most teams land on is integration tests that hit the real site on every commit. They are slow. They flake for reasons that have nothing to do with your code. They get muted within a month. And running them on a schedule against somebody else's server is not testing, it is load.
The other popular compromise, replayed HTTP mocks, fails more quietly: it tests your code against a snapshot of a web that no longer exists, which is precisely the failure mode it cannot see.
And the original non-strategy — no tests, the dashboards will tell us — conflates detection with correction. A dashboard can tell you it broke. It has never once told anybody that the fix is safe to ship.
The objection I keep hearing
"The site is too dynamic for fixtures." Then stop testing the markup and test the invariants instead: every item has a non-empty title, every price parses to a number with a currency attached, item counts stay inside sane bounds, every record matches the schema you promised downstream. These hold no matter how much the HTML churns, and they are exactly the guarantees the rest of your system actually depends on.
None of this is exotic. Layer 1 is a recording script and ordinary unit tests. Layer 2 is a loop over a handful of mutations. Layer 3 is a cron job you already know how to write. Together it is about two days of work, and it moves every scraper change from "deploy and pray" to "red test, green test, ship."
So next time you merge a selector change, ask the only question that matters: what test would have failed? If the honest answer is "none, we would find out in the tables," then your test folder is not empty because scrapers cannot be tested. It is empty because nobody has gotten around to the two days yet.
How do you test yours — or do you also have a folder with one skipped test in it?