OpenAI supports robots.txt signals
The story so far
OpenAI will respect the rules set in the robots.txt files operated by website owners to influence the data scraping behavior of AI models on websites, helping website operators control whether their website content is used for AI training. OpenAI, Google, Microsoft, and Anthropic jointly support new website content scraping opt-out standards, allowing publishers to opt out of AI scraping by adding a shared 'policy token' signal in the robots.txt file. This aims to enable website operators to more effectively express their preferences for AI training, allowing content publishers to uniformly declare their preferences for AI model training to multiple AI companies through the new standard in one go, rather than setting rules for each company individually. Anthropic has adopted it first, OpenAI plans to implement it in the coming months, and Google and Microsoft are expected to support it next year.
AI-written from the reports · Updated 17 days ago
Developments
2 developments
- 1 Nov, 22:19 · 1 reportsAI data scraping exit standardsITHome · AI:OpenAI, Google, Microsoft, and Anthropic jointly promote AI data scraping exit standards
- 30 Oct, 17:30 · 2 reportsHonoring shared robots.txt signalOpenAI News:Making it easier for websites to express AI training preferences
Coverage timeline
Follow the coverage to see every side of the story.
- ITHome · AIOpenAI, Google, Microsoft, and Anthropic jointly promote AI data scraping exit standards
Content publishers can use the new standards to choose to reject AI scraping website content without setting rules for each company; the standards are proposed by Anthropic and integrated into robots.txt files, Anthropic has adopted them first, OpenAI plans to implement them in the coming months, and Google and Microsoft are expected to support them next year.
- The Verge · AIThe biggest names in AI are backing a new way to opt out of scraping
OpenAI, Google, Microsoft, and Anthropic will adopt new standards that allow publishers to opt out of AI scraping by adding a shared 'policy token' signal in their robots.txt file, thereby controlling whether their website content is used for AI training.
- OpenAI NewsMaking it easier for websites to express AI training preferences
OpenAI will support website operators in conveying AI training preferences to providers participating in the service through robots.txt signals.