Most of us here have shipped a scraper that worked fine on Monday and returned an empty list on Tuesday. This is the debugging order I settled on after enough of those afternoons, written out so the next person spends ten minutes on it instead of four hours.
The Inspector Shows You a Different Document
When your selector matches nothing, the instinct is to blame the selector. Open DevTools, find the element, copy the path, paste it in, still nothing.
The problem is that DevTools shows the DOM after every script on the page has run. Your scraper, if it is requests or httpx, sees the raw HTML the server sent and nothing else. Those two documents are often not the same document.
So the first move is not a better selector. It is view-source, or a print of response.text. If the value you want is not in there, it was never going to be, and every selector you try after that is wasted time.
Static and Rendered Are Two Different Jobs
Once you know which kind of page you have, the tool choice makes itself.
If the data is in the response body, requests plus BeautifulSoup is the right answer, and it runs roughly an order of magnitude faster than anything that starts a browser. Swapping in lxml as the parser is worth it once volume grows.
If the data arrives after render, you need something that executes JavaScript. Playwright is the usual pick for new work, mostly because auto-waiting removes the sleep-and-hope pattern that makes older Selenium suites flaky.
There is a middle path people miss. Plenty of so-called JavaScript sites are calling a JSON endpoint you can call yourself. Open the network tab, filter to XHR, and look at what the page is fetching. Hitting that endpoint directly is faster and far more stable than driving a browser to read the rendered result.
Scrapy Solves Scale, Not Parsing
Scrapy usually gets recommended as a BeautifulSoup alternative, which muddles the decision. It is a crawling framework: scheduling, concurrency, retries, throttling, item pipelines. On a fifty page job those are machinery you maintain for no gain. On fifty thousand pages they are the reason the job finishes at all.
What Actually Breaks in Production
Three things, in roughly this order.
Layouts change, so a selector anchored to a generated class name dies on the first redesign, while text and attribute anchors survive much longer.
Rate limits arrive without warning, so backoff and a real delay belong in version one instead of after the ban.
Silent failures are the expensive ones. A scraper that returns zero rows and exits clean will feed an empty file into whatever is downstream, quietly, for weeks. An assertion on expected row count is three lines and catches most of it.
Takeaway
Find out where the data lives before you choose a library. Static pages want requests and BeautifulSoup, rendered pages want Playwright or the JSON endpoint sitting behind them, and Scrapy is for when volume rather than parsing is the problem.
The longer version, with the full library comparison, pagination handling and export formats, is in Web Scraping with Python: The Complete Guide.
