Cloudflare has announced a new "Disallow AI Training" setting designed to help website owners navigate the tradeoff between AI training and search engine discoverability. This feature specifically addresses "mixed-use crawlers"—bots that serve both search engine indexing and AI model training. Previously, blocking such crawlers meant risking a site's visibility in search results.
Cloudflare Introduces "Disallow AI Training" Setting to Protect Content While Maintaining Search Visibility
The new setting works by publishing a Disallow: AI Training directive in the site's robots.txt file. According to Cloudflare, major operators including Apple, Google, and Microsoft have either implemented or committed to honoring this setting. This initiative is part of Cloudflare's "Accountable" designation, which recognizes crawler operators that provide transparency and respect publisher choices regarding content usage.
While the new setting addresses training, Cloudflare noted that controls for AI summaries are still evolving. The company's goal is to provide granular control, allowing site owners to decide how much of their content is used for different purposes. These new controls are available to all Cloudflare customers across all plans through the domain Security Settings.
Sources
- Cloudflare: Stay discoverable in search while disallowing AI training （Hacker News Frontpage, 2026-09-16）