www.seroundtable.com AI Giants Introduce Unified Opt-Out for Content Scraping
Article Content
- •AI companies introduce a unified opt-out policy for content scraping.
- •Publishers can now set AI training preferences in a single robots.txt file.
- •Implementation will be gradual, with full support expected by 2027.
OpenAI, Google, Microsoft, and Anthropic announced a new shared policy token that allows publishers to opt out of having their content scraped for AI training. This initiative aims to simplify the process for publishers by enabling them to set their AI training preferences in a single robots.txt file, rather than individually for each AI company. The proposal, supported by the Internet Engineering Task Force (IETF), is designed to address concerns from publishers about unauthorized scraping by AI crawlers. Microsoft will provide additional granular controls for publishers, allowing them to opt out of AI training or grounding. The implementation of this policy is expected to roll out gradually, with Google planning to adopt it in 2027 and OpenAI in the coming months. This move comes amid increasing pressure from publishers and regulators for clearer guidelines on content usage by AI models.
Ask AI about this cluster
Answers cite the sources they use