Skip to content
Live
Trending storyPast story

OpenAI supports robots.txt signals

3 reports3 outlets17 days agoUpdate

The story so far

AI overview

OpenAI will respect the rules set in the robots.txt files operated by website owners to influence the data scraping behavior of AI models on websites, helping website operators control whether their website content is used for AI training. OpenAI, Google, Microsoft, and Anthropic jointly support new website content scraping opt-out standards, allowing publishers to opt out of AI scraping by adding a shared 'policy token' signal in the robots.txt file. This aims to enable website operators to more effectively express their preferences for AI training, allowing content publishers to uniformly declare their preferences for AI model training to multiple AI companies through the new standard in one go, rather than setting rules for each company individually. Anthropic has adopted it first, OpenAI plans to implement it in the coming months, and Google and Microsoft are expected to support it next year.

AI-written from the reports · Updated 17 days ago

Developments

2 developments

  1. 1 Nov, 22:19 · 1 reports
    AI data scraping exit standards
    ITHome · AI:OpenAI, Google, Microsoft, and Anthropic jointly promote AI data scraping exit standards
  2. 30 Oct, 17:30 · 2 reports
    Honoring shared robots.txt signal
    OpenAI News:Making it easier for websites to express AI training preferences

Coverage timeline

Follow the coverage to see every side of the story.

1 Nov
  1. ITHome · AI
    OpenAI, Google, Microsoft, and Anthropic jointly promote AI data scraping exit standards

    Content publishers can use the new standards to choose to reject AI scraping website content without setting rules for each company; the standards are proposed by Anthropic and integrated into robots.txt files, Anthropic has adopted them first, OpenAI plans to implement them in the coming months, and Google and Microsoft are expected to support them next year.

31 Oct
  1. The Verge · AI
    The biggest names in AI are backing a new way to opt out of scraping

    OpenAI, Google, Microsoft, and Anthropic will adopt new standards that allow publishers to opt out of AI scraping by adding a shared 'policy token' signal in their robots.txt file, thereby controlling whether their website content is used for AI training.

30 Oct
  1. OpenAI News
    Making it easier for websites to express AI training preferences

    OpenAI will support website operators in conveying AI training preferences to providers participating in the service through robots.txt signals.