Article 349: Four AI Labs Agreed on One Line of robots.txt, and It Only Takes Full Effect in 2027

Yesterday OpenAI, Google, Microsoft and Anthropic said they will honour a single shared training signal instead of forcing every site to list each company's crawler by name. The "policy token" comes from a draft still circulating at the IETF and was proposed by Kevin Kelley, a standards lead at Anthropic.

The details are where the story is.

Support is real, delivery is dated

Anthropic already reads the token. Google and Microsoft have set 2027. OpenAI has named no month.

That means a site owner reading yesterday's headlines, adding one line, and assuming they are out of every training set would be wrong for the largest lab of the four, and wrong for two others until some point next year. A standard adopted partially and on staggered dates is not a standard yet. It is a schedule, and schedules slip.

Microsoft's version is the interesting one

Alongside the training signal, Bing added a page-level grounding setting - grounding being the use of your page to write an AI answer, with a link back to it.

That splits what used to be one decision into two. You can refuse training and still be cited in Copilot. You can refuse training and limit grounding to Microsoft's own products, keeping third-party answer engines out. Microsoft is unusually blunt about the trade: a brand that blocks grounding shows up less in those AI answers.

Which admits something worth naming. For a lot of sites, being summarised is now the traffic. Refusing it is a choice between being quoted for free and not being seen.

Two things this does not cover

The token governs what crawlers collect from your site. It says nothing about what a person types into an assistant, and it says nothing about content already ingested. Microsoft also stops treating its old NoArchive and NoCache tags as AI preferences after 1 March 2027, so anyone relying on those tags has a migration to do.

Key Takeaway: a shared opt-out is genuinely better than nine crawler names in a text file, and it is the first sign the labs accept that "we respect robots.txt" needed to stop being a per-company negotiation. But read the dates, not the announcement. As of today, exactly one of the four honours the line.