Cloudflare Implements Controls Against AI Scraping Bots

Cloudflare announced new measures to block mixed-use crawlers from accessing ad-supported websites, effective September 15, 2026. This decision aims to give publishers more control over their content, particularly against crawlers that scrape data for AI training. Major tech companies like Google, Apple, and Microsoft may be impacted, as their crawlers often combine indexing and data harvesting. The new controls will apply to all customers, including those on the free tier, and will replace the previous 'block all' approach. Cloudflare's CEO emphasized the need for a sustainable ecosystem as non-human traffic dominates the internet. The changes will allow site owners to explicitly permit or deny access to their content for AI training purposes. This initiative reflects growing concerns among publishers about compensation for their content used in AI models.

Key Points: • Cloudflare will block mixed-use crawlers from ad-supported sites starting September 15, 2026. • The new controls allow publishers to manage automated traffic based on specific AI use cases. • Major tech companies' crawlers may be affected, as they often scrape data for AI training.