Most of us here have written a scraper or a browser test suite that worked beautifully until someone shipped a redesign and every selector went stale overnight. This is a look at what actually changes when an agent reads the page instead of replaying a recorded path, and where that tradeoff stops paying for itself.
Why Recorded Steps Break
Traditional web automation targets elements by a fixed path. A class name, an xpath, a nth-child chain. That works right up until the markup moves, and then the script is not wrong in an obvious way, it is confidently clicking nothing. Anyone who has maintained a suite across a design refresh knows the failure is rarely a crash. It is a green run that tested an empty page.
An agent driving the same browser reasons about the page as it exists at that moment and finds the element by its purpose rather than its address. The login button is whatever currently behaves like the login button. That is the entire difference, and it is why the same agent can complete a task on a site it has never seen.
Three Ways An Agent Perceives A Page
Under the surface there are only a few representations in play, and most tool differences come down to how they mix them.
The DOM gives precise targeting and full text, and it drowns you in presentational noise while hiding the meaning that only layout carries. The accessibility tree, the semantic view browsers already build for screen readers, strips the clutter and labels each element by role and purpose, which usually makes it the cleanest input a model can get. And screenshots with numbered overlays on the interactive elements let a vision model say element 7 instead of describing a position in prose.
The setups that hold up in production combine them. Structure for exact targeting, the visual pass for understanding and disambiguation. A deeper breakdown of how agents control and perceive a browser is in this guide to AI browser automation, including where the frameworks and protocols underneath fit.
The Part That Still Bites
Perception does not save you from timing. Most of the modern web arrives nearly empty and fills itself in after running scripts, so an agent that reads too early acts on a page that does not exist yet. It sees a skeleton, decides confidently, and clicks a placeholder. This is the same wait-for-content problem the testing world has fought for a decade, and adding a language model does not remove it. It just makes the wrong answer more fluent.
When A Plain Script Still Wins
The honest limit is cost and determinism. An agent run costs model tokens and takes longer, and it can make a different choice on two identical pages. If a clean API exists, call it. If the target is one stable page you control, a plain script is cheaper, faster and easier to debug at 3am.
Browser agents earn their keep on the messy end: many sites with different layouts, no programmatic interface, and a task description that survives a redesign better than a selector does. That is a real category of work, and it is worth reaching for exactly there rather than everywhere.