Search engines learned about websites from robots.txt and sitemap.xml. AI agents need more, because they don't just read pages; they take actions. Between 2024 and 2026 a set of new files grew up that describe a site to agents, list its capabilities and explain how to connect. A few are formal standards, some are proposals with real adoption and some are experiments, and this guide says which is which.
At the root there's robots.txt for permissions, the sitemap for completeness, llms.txt and llms-full.txt for a curated summary and full text, and agents.txt, a newer and less settled convention I'd treat as a readable pointer file rather than a contract.
Under /.well-known/, reserved for site metadata by RFC 8615, the entries that matter are the A2A agent card, an MCP server card, OAuth protected-resource metadata (RFC 9728), authorisation server metadata, an API catalogue (RFC 9727) and security.txt (RFC 9116). I include a minimal agent card and a warning: a card that promises skills your endpoint doesn't have is worse than no card.
Discovery also works from any page through rel=alternate links and Link headers. I finish with the order I'd add everything in, starting with robots.txt and sitemaps and ending with agent cards and an API catalogue.
Key takeaways
- Agents need discovery files beyond robots.txt and sitemaps
- Know which files are standards and which are proposals
- The /.well-known/ path holds agent cards, OAuth metadata and security.txt
- Keep agent cards accurate and minimal
- Add discovery links in page heads and HTTP headers too
Related on BuiltToWinWeb: the AI agent-readiness checklist
Read the full guide on BuiltToWinWeb: Every agent discovery file explained
A summary by Jacob Campbell of an article first published on built2winweb.com on October 5, 2026.
