Website owners may soon be able to tell several AI companies "don't train on my content" with one entry in robots.txt instead of a growing list of crawler-specific rules. On October 30, 2026, Google, OpenAI and Microsoft each announced plans to honor a shared AI-training preference signal. It comes from an Internet-Draft written by Kevin Kelley, a standards lead at Anthropic, and builds on work in the Internet Engineering Task Force's AI Preferences (AIPREF) working group.
The mechanism adds no new robots.txt directive. A site writes an ordinary rule group, but instead of naming a real crawler on the User-Agent line it names a "policy token," __usage_ai-training, and uses the familiar Allow and Disallow lines to mark which paths it does or does not want used for generative AI training. The draft's own example blocks training across a whole site except a /press-kit/ folder. Parsers that don't recognize the token skip it as they would any unknown user agent.
Today, a publisher that wants to stay out of training data has to track each company's crawler and control: GPTBot for OpenAI, Google-Extended for Google, and others. "Previously, website operators would identify individual crawlers and maintain separate rules for each and every provider," OpenAI wrote in its announcement. "Having a shared signal across multiple providers reduces that work for operators."
The companies agreed on the signal but not on a start date. OpenAI says it aims to implement it "in the coming months," and that its existing crawler controls will continue to apply. Google's Search Central post, by Search Advocate John Mueller, says it aims to support the control in 2027; Google's crawler documentation, updated the same day, says early 2027, The Next Web reports. Microsoft says it will honor the preference from February 1, 2027, according to its Bing blog post as reported by The Next Web. Kelley wrote on the IETF mailing list that Anthropic has started implementing support, without giving a launch date.
The details matter for publishers. Google says that when a site uses the token to opt out and does not name Google-Extended, it will treat that as a Google-Extended disallow. Google-Extended also governs whether Gemini Apps can ground answers in a site's pages and link to them, so at Google an opt-out from training also means an opt-out from that Gemini grounding. Google Search ranking and inclusion are not affected, and existing Google-Extended rules still take precedence. Microsoft, according to The Next Web and Search Engine Journal, is adding a separate page-level setting for AI grounding, retiring its older NoArchive and NoCache tags, and says the new preferences "do not establish licensing or payment arrangements."
The limits are just as important. The draft describes the token as a statement of preference about how content is used, not a crawling block. A site that has no such group has expressed no preference, and a blanket User-Agent: * rule does not count. Private agreements between a site and an AI company can override it. As Engadget notes, the signal also cannot undo scraping that has already happened. And the draft is an individual submission: the IETF has not endorsed it, and it has no formal standing in the standards process yet.
Even so, it is one of the clearest signs yet of the largest AI developers agreeing on a common, machine-readable way for publishers to set terms for training. Google says the approach is consistent with the EU's Code of Practice for general-purpose AI models, which recognizes robots.txt as a primary way to express such preferences. The harder questions, including whether publishers get paid and how AI products cite and send traffic to their work, remain outside what a robots.txt line can settle.