Skip to content
AI Giants Introduce Unified Opt-Out for Content Scraping

AI Giants Introduce Unified Opt-Out for Content Scraping

First seen 30 Oct 2026, 21:14 UTC • •

Article Content

Browse articles
ThreatCluster AI
ThreatCluster •October 31, 2026 at 21:13 UTC
  • •AI companies introduce a unified opt-out policy for content scraping.
  • •Publishers can now set AI training preferences in a single robots.txt file.
  • •Implementation will be gradual, with full support expected by 2027.

OpenAI, Google, Microsoft, and Anthropic announced a new shared policy token that allows publishers to opt out of having their content scraped for AI training. This initiative aims to simplify the process for publishers by enabling them to set their AI training preferences in a single robots.txt file, rather than individually for each AI company. The proposal, supported by the Internet Engineering Task Force (IETF), is designed to address concerns from publishers about unauthorized scraping by AI crawlers. Microsoft will provide additional granular controls for publishers, allowing them to opt out of AI training or grounding. The implementation of this policy is expected to roll out gradually, with Google planning to adopt it in 2027 and OpenAI in the coming months. This move comes amid increasing pressure from publishers and regulators for clearer guidelines on content usage by AI models.

Start a free Starter trial for enhanced analysis

Ask AI about this cluster

Updated 15d ago How this analysis works

Timeline

2026-10-30
AI giants announce shared opt-out policy
OpenAI, Google, Microsoft, and Anthropic revealed a new policy token for publishers to manage AI scraping preferences.
Theverge
2026-10-31
Further details on implementation shared
The gradual rollout of the new opt-out feature was confirmed, with Microsoft offering additional controls for publishers.
Feeds.4Sysops

More articles in this cluster (6)

Common questions

How does the new opt-out policy work?
Publishers can use a shared policy token in their robots.txt file to indicate their AI training preferences.
When will this policy be fully implemented?
Google plans to implement the policy in 2027, while OpenAI will adopt it in the coming months.
What additional controls will Microsoft provide?
Microsoft will offer granular controls allowing publishers to opt out of AI training or grounding.