robots.txt only decides whether a crawler may fetch a path. It's silent on what happens next: whether the content appears in a search result, gets quoted in an AI answer or is used to train a model. Two newer mechanisms address that, and this guide explains both and is straight about their limits.
Content Signals, introduced by Cloudflare in September 2025, adds a Content-Signal line to robots.txt with three settings: search, ai-input (using content in a real-time AI answer) and ai-train. Each is yes or no, and a missing key means no stated preference. The split matters because search and AI answers send visitors and citations, while training only helps you indirectly, and robots.txt alone can't express "answer questions with my content but don't train on it" when one crawler serves several purposes. On this site I publish yes to all three.
RSL (Really Simple Licensing), also announced in September 2025, defines a machine-readable licence stating what AI systems may do with content and on what terms, from free with attribution to pay per crawl. We publish one here as an experiment.
The limits are important. Neither is enforcement, neither reaches content already collected, and real enforcement lives in CDN bot management and verified-bot programmes. I finish with a short list of what to publish and how to keep it consistent with your user-agent rules.
Key takeaways
- robots.txt controls fetching, not later use
- Content Signals covers search, ai-input and ai-train
- RSL publishes machine-readable licence terms
- Both state preferences; neither enforces them
- Make sure your CDN isn't overwriting your signals
Related on BuiltToWinWeb: Cloudflare's AI crawl controls
Read the full guide on BuiltToWinWeb: Content Signals and RSL explained
A summary by Jacob Campbell of an article first published on built2winweb.com on October 5, 2026.
