🔍 Read the full analysis: How To Set Your AI Training Preferences With The Shared Opt-Out on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
OpenAI, Google, Microsoft and Anthropic have said they will support a shared robots.txt signal asking that website content not be used to train generative AI. The rule is voluntary, takes effect on different schedules, and does not block crawling, remove content from existing models or opt out of AI answers based on live web content.
OpenAI, Google, Microsoft and Anthropic said on October 30 that they will honor a shared instruction in website robots.txt files asking that content not be used to train generative AI models. The proposal could spare publishers from maintaining separate training opt-outs for each participating company, but it remains a voluntary signal, and the companies have set different implementation timelines.
The proposed rule uses a shared user-agent label, User-Agent: __usage_ai-training, followed by Disallow: /. A site administrator can place the rule in robots.txt to state a preference against training use. The signal is described in an individual Internet-Draft titled “Machine-Readable Training Preference Signal,” published on October 30 by Kevin Kelley, an Anthropic standards lead.
The companies’ plans are not identical. Anthropic says it has already started implementing support, though it says support across its services will take time. OpenAI says it aims to honor the signal in the coming months. Google aims to support it in 2027, with its crawler documentation specifying early 2027. Microsoft says it will honor the shared training preference from February 1, 2027, as it crawls websites.
Microsoft also announced a separate page-level preference for AI Grounding, covering web content used to inform answers in Bing, Copilot and Microsoft 365. Microsoft says both new preferences take effect on February 1, 2027. Its older NoArchive and NoCache controls will stop counting as preferences for training or grounding after March 1, 2027. The companies have not said that the shared rule will prevent crawling or affect ordinary search eligibility.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
The proposal could make it easier for publishers to communicate a training preference across several major AI providers. Under the previous approach, a site owner might need distinct instructions for services such as OpenAI’s GPTBot, Google-Extended and Anthropic’s ClaudeBot, as well as Microsoft-specific page controls. A common rule may reduce that administrative burden once companies implement it.
But the practical effect depends on providers choosing to honor the request and applying it across their services. Robots.txt is not an access-control system: the signal expresses a preference; it does not technically prevent a company from retrieving a page. Nor does the rule promise payment, traffic or removal of material already used in training. Microsoft said that neither choice guarantees inclusion, traffic or payment.
Publishers also need to distinguish training from the use of current web pages to generate AI answers. The shared token addresses foundational model training, not every use of a page in an AI product. Microsoft’s separate grounding control highlights that these are different choices. Organizations deciding how to manage their content may need to set preferences for training, AI answers and search separately.
Products Worth Considering
Samsung Galaxy Tab S10 FE 128GB Wi-Fi Android Tablet, Blue
Bose QuietComfort Headphones (2nd Gen), Wireless Noise-Cancelling Over-Ear Headphones, Black
LG 34GX90SA-W 34-inch Ultragear WQHD OLED Curved Gaming Monitor
As an affiliate, we earn on qualifying purchases.
How the Rule Reached Companies
The shared token comes from an individual Internet-Draft associated with the IETF’s AI Preferences working group. The IETF develops internet standards, but publication of a draft does not make it an adopted standard. The draft’s own status is limited: it has no formal standing in the IETF standards process.
At a working-group meeting on October 21, one chair said, “We haven’t declared consensus on anything yet.” That means the announcements describe company plans around a proposed signal, not a finalized internet-wide requirement. The draft was written by an Anthropic standards lead, and its acknowledgments list contributors from technology companies, publishers and other organizations. Those affiliations provide context, but do not establish that every named organization has committed to support it.
Meta’s communications chief, Andy Stone, said the company supports publishers’ ability to express preferences and called a common signal an important step. He did not say whether Meta will honor the rule. The source material also reports no statement from Apple, Amazon, xAI or Perplexity committing to support it. The scope of adoption beyond the four announcing companies remains unsettled.
““has already started implementing support for this draft””
— Anthropic
Adoption and Coverage Remain Open
Several details remain unresolved. The Internet-Draft has not been adopted as an IETF standard, and the participating companies have not all begun implementation. Anthropic’s public crawler help page still described its existing crawler controls as of October 31. The precise timing and coverage of each provider’s rollout may change as work proceeds.
It is also not clear whether Meta or other AI companies will honor the shared token. Even among companies that have announced support, the signal is limited to specified uses: it does not automatically block crawling, cover AI answers based on live web content, or undo training that has already taken place. Publishers should check each provider’s documentation rather than assume that one line governs every product or use.
Rollouts Begin on Separate Schedules
Website owners can add the shared training preference now if they want to state that choice, but should keep existing provider-specific rules in place until each company confirms the relevant controls are active. Anthropic says its rollout is underway; OpenAI has pointed to the coming months; Google is targeting early 2027; and Microsoft’s stated start date is February 1, 2027.
Sites using Microsoft’s NoArchive or NoCache controls should review Microsoft’s documentation before the March 1, 2027 change, particularly if they also want to express a separate preference about AI Grounding. Publishers can also follow updates to the Internet-Draft and company crawler pages. The next meaningful test will be whether announced support becomes working, clearly documented behavior across services—and whether more providers join.
Key Questions
What line does the shared AI training opt-out use?
The proposed robots.txt group is User-Agent: __usage_ai-training followed by Disallow: /. It expresses a preference against using the site’s content for generative AI training.
Does the rule stop AI companies from crawling a website?
No. The signal communicates a preference; it is not a technical barrier that prevents a crawler from accessing pages. The announcements describe it as voluntary.
Does opting out cover AI answers or remove existing training data?
No. The shared signal concerns training, not AI answers generated using live web content. It also does not remove material from models that have already been trained.
When will the companies honor the signal?
Anthropic says implementation has started, OpenAI aims for the coming months, Google targets early 2027, and Microsoft says it will begin on February 1, 2027. These are company timelines, not a shared mandatory start date.
Should website owners remove their existing crawler rules?
Not yet based on the announcements described here. Existing company-specific controls continue to apply, and keeping them until providers confirm their new support can help avoid gaps during staggered rollouts.
Source: ThorstenMeyerAI.com
