robots.txt: one shared AI training signal, and the decisions it forces on merchants

On 30 October 2026, three announcements landed on the same day. On its Search Central blog, Google said it will support a shared AI training control in robots.txt; OpenAI published a post along the same lines; Microsoft laid out its timeline for Bing. The common thread is a single token, defined in an IETF Internet-Draft written at Anthropic and published that same day, which every participating provider commits to reading the same way.

What actually changes

Until now, a merchant who wanted to keep content out of model training had to maintain a list of proprietary tokens: Google-Extended at Google since 2023, GPTBot at OpenAI, the NoArchive and NoCache tags at Microsoft, and one more line for every new player. The IETF draft replaces that with a single policy token, __usage_ai-training, written like any ordinary robots.txt group:

• User-Agent: __usage_ai-training
• Disallow: /
• Allow: /press/

Read by a participating provider, that group expresses a preference against training across the whole site, except the press folder. The grammar is RFC 9309: the same path rules, the same longest-match precedence, no new directive to learn.

Three details matter to an architect.

• It is not a crawl rule. The token never appears in a bot’s request header; it describes a use, not an access right. To limit crawling, the Googlebot, Bingbot or GPTBot groups remain the right layer.
• The wildcard group does not count. A robots.txt that only blocks the basket and the customer account under the star expresses no training preference at all. The token has to be named.
• No token, no preference. Providers then fall back on their usual rules, and a proprietary token keeps its own meaning.

Training is not visibility

This is the main trap for an online retailer. The token covers one thing only: training, which the IETF defines as using content to modify the learned parameters of a generative model. It says nothing about classic search, about answers generated from the web (what providers call grounding), or about shopping agents reading your catalogue live.

Providers nevertheless tie the two together, each in its own way. Google states that a site opting out of training through the shared token without naming Google-Extended will be treated as if it had disallowed Google-Extended; yet only sites that explicitly allow Google-Extended remain eligible for answers and clicks through grounding in the Gemini apps. Refusing training at Google can therefore cost presence in Gemini. Microsoft, by contrast, separates the two uses: training is set in robots.txt or per page, grounding per page only, with an intermediate option, DisallowExtended, that reserves your content for Microsoft experiences (Copilot, AI Mode, Microsoft 365) and withdraws it from third-party services. Microsoft warns that refusing grounding will reduce brand visibility in its AI experiences, without affecting classic Bing results.

For a merchant, the question is therefore not “block AI or not” but: which content carries training value you do not want to give away (buying guides, comparisons, photos, reviews), and which presence in assistants you do not want to lose (product pages, stock, prices)?

The timeline

• Google aims to support the control in 2027, alongside the other providers; Google-Extended keeps working in parallel.
• OpenAI announces a rollout in the coming months and confirms that its existing controls remain respected.
• Microsoft applies the new preferences from 1 February 2027. On 1 March 2027, NoArchive and NoCache stop being read as training or grounding preferences: sites relying on them must publish page-level preferences before that date, and a transitional grounding setting opens in Bing Webmaster Tools on 15 December 2026. Both preferences default to Allow.
• On the standards side, the text is still a draft that expires on 3 May 2027, and discussions on the rest of the IETF vocabulary continue. Google notes that the approach is consistent with the European code of practice for general-purpose AI models, which recognises robots.txt as a primary way to express these preferences.

What to do on a Magento or Adobe Commerce store

1. Inventory what exists. On Magento 2 and Adobe Commerce, robots.txt is edited in the admin, at website scope, in the design configuration; in a multi-site setup each domain has its own. Also record the meta robots tags and X-Robots-Tag headers set by the theme, SEO modules and the CDN: they often contradict each other.
2. Decide per zone, not per site. Catalogue, editorial content, B2B area, media: each zone deserves its own answer, written with marketing and legal, then versioned with the code.
3. Remove nothing before 2027. The Google-Extended, GPTBot and similar lines remain valid; the shared token is added, it replaces nothing, and a provider may treat its own token as more specific.
4. Handle decoupled storefronts. On a headless, Edge Delivery or Next.js front end, robots.txt is no longer served by Magento: the file and the headers belong to the front-end team and the CDN. There must be a single owner.
5. Measure. Log AI bot visits before and after, and track visits coming from assistants; it is the only way to estimate the real cost of a refusal.

The shared token makes the preference easier to write; it does not decide for you. The trade-off between protecting content and being present in assistants is an architecture and leadership decision, to be made now, while providers are leaving time to prepare it.