Google Shared AI Training Control in robots.txt (2026): Policy Token, Google-Extended & What to Do Now

Short answer: On October 30, 2026, Google announced plans to honor a shared robots.txt AI training policy token (IETF AIPREF work) so publishers can express training preferences once across participating AI providers. If you opt out with that token and do not explicitly set Google-Extended, Google plans to treat it like disallowing Google-Extended. OpenAI says it will honor the common robots.txt training signal in the coming months. Google aims for joint support in 2027 — this is not live as a Google Search / AI Overviews crawl switch today.

This guide separates the new shared training preference from Googlebot, AI Overviews / AI Mode visibility, and the per-crawler rules you already maintain. Official sources: Google Search Central (John Mueller, Oct 30, 2026), OpenAI’s training-preferences post, the IETF draft for the training preference signal, and industry context from Search Engine Land.

What Google announced on Oct 30, 2026

Google’s Search Central blog says the company plans to support a shared AI training control in robots.txt, based on work from the IETF AIPREF working group. Instead of listing every AI product token separately for a broad “do not train” preference, site owners will be able to use a shared policy token to set preferences across participating providers.

• Opt-out interpretation: When a site uses the policy token to opt out of training and Google-Extended is not explicitly specified, Google plans to interpret that as if the publisher had disallowed Google-Extended.
• Company-specific controls stay: Existing controls such as Google-Extended continue to work alongside the joint signal — you can set a broad preference, then tailor Google (or others) individually.
• Gemini Apps eligibility: Sites that specifically allow Google-Extended remain eligible to appear in responses and receive clicks via grounding in Gemini Apps.
• Timeline: Google aims to support the new control along with other AI providers in 2027, with further updates as the IETF work progresses.
• Policy context: Google notes consistency with the EU Code of Practice for GPAI Model Providers under the EU AI Act, which recognizes robots.txt as a primary machine-readable preference method.

What OpenAI announced the same day

OpenAI said it will honor a common robots.txt signal for foundational generative AI training preferences. Goals and caveats from that post:

• One shared signal reduces the need to maintain separate training rules for every provider.
• Implementation is planned in the coming months (OpenAI’s wording), with a follow-up when details ship.
• Existing crawler controls in robots.txt remain and continue to be respected — the shared signal does not replace them.
• The effort draws on the IETF AI Preferences (AIPREF) working group; OpenAI encourages other providers to honor the signal.

Search Engine Land reports that Bing and Anthropic were also described as adopting the IETF proposal alongside Google and OpenAI. Treat vendor-by-vendor rollout dates as independent — only Google’s 2027 aim and OpenAI’s “coming months” are in those companies’ own posts.

The IETF policy token: __usage_ai-training

The Internet-Draft draft-kelley-aipref-training-preference-signal (Kevin Kelley / Anthropic, 30 Oct 2026) describes binding usage preferences to paths with an ordinary robots.txt group keyed by a policy token on the User-agent line. Example from the draft:

User-agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That expresses a preference against AI training for most paths, and in favor for /press-kit/. Important design points for publishers:

• No new robots.txt grammar: Allow/Disallow and longest-match rules stay as in RFC 9309.
• Independent from crawl rules: Training preferences are orthogonal to traffic-management groups for Googlebot, GPTBot, etc. Legacy parsers that do not understand the token ignore the group.
• Wildcard User-agent: * does not match the policy token — you must name __usage_ai-training explicitly.
• AI Training definition (via AIPREF vocabulary): using an asset to modify learned parameters of a generative AI model — not exclusive non-generative tasks, and not classification/ranking/scoring alone.

Policy token vs Google-Extended vs per-bot Disallow

Do not collapse three different levers into one. Pair this table with our AI crawlers allow/block cheat sheet and Google’s own crawler overview docs.

Control	What it mainly governs	Status for Google (as of Oct 30, 2026)	Does it block Google Search / AI Overviews?
Shared policy token (__usage_ai-training)	Cross-provider generative AI training preference	Announced; Google aims for support with peers in 2027	No — training preference, not Googlebot
Google-Extended	Google’s company-specific AI training / Gemini Apps grounding eligibility	Live since 2023; still authoritative when set explicitly	No — separate from Googlebot
Per-bot Disallow (GPTBot, ClaudeBot, Bytespider, …)	That product’s crawl / fetch rules	Live today (honor varies by vendor)	Only if you Disallow Googlebot (usually a bad idea for SEO)
Search Console generative AI control	Whether your site can appear in Search generative AI features (Overviews / AI Mode)	Live product setting	Affects Search generative visibility — see our GSC control guide

Google’s announcement is explicit: allowing Google-Extended keeps you eligible for Gemini Apps grounding and clicks. Opting out via the future policy token (when Google-Extended is unspecified) is planned to behave like disallowing Google-Extended — which is a training / Gemini Apps decision, not a Search ranking switch. For Overviews CTR and halo effects, see the Shepard opt-out / halo study notes; for invite-only payments when content shapes answers, see the AI contribution pilot.

What to do now vs what to wait for in 2027

Do now (low regret)

1. Inventory robots.txt for Google-Extended, GPTBot, OAI-SearchBot, ClaudeBot, and other tokens you already manage — keep the sheet you use with our crawler listicle.
2. Decide Google-Extended intentionally. Allow if you want Gemini Apps grounding eligibility; Disallow if you want to opt out of Google’s AI training control as it exists today. Do not confuse that with Googlebot.
3. Keep Search Console generative AI control on Include unless you have a deliberate legal/licensing opt-out — Exclusion is a Search product switch, not the IETF training token (walkthrough).
4. Ship citation-ready pages that still win retrieval: clear answers, unique evidence, honest schema. Training preferences will not replace content quality — see Google’s generative AI SEO guide and our get-cited checklist.
5. Watch the IETF draft and vendor blogs for syntax freezes and honor dates before mass-deploying __usage_ai-training as if it were already enforced everywhere.

Wait / prepare for 2027

• Expect Google’s shared policy-token support to land with other providers on a 2027 timeline (Google’s own words) — plan a robots.txt change window then, not a panic edit tonight.
• When you adopt the token, decide whether to also keep an explicit Google-Extended group. Explicit company rules remain the way to “pick and choose” after a broad opt-out.
• Document path exceptions (press kits, docs you want trained) with Allow lines under the policy-token group, matching the draft’s press-kit example.
• Align legal / licensing review with product SEO so training opt-outs do not accidentally Disallow Googlebot or flip Search generative controls.

Publisher checklist: robots.txt training preferences

1. Goal clarity: Are you optimizing for Search citations, Gemini grounding clicks, model-training refusal, or all three? They use different levers.
2. Googlebot: Leave crawlable for pages you want in Search / AI Overviews / AI Mode.
3. Google-Extended: Set Allow or Disallow explicitly based on Gemini Apps + Google training preference.
4. Other bots: Keep per-vendor groups current; the shared token is additive, not a full replacement on day one.
5. Policy token (when supported): Add User-agent: __usage_ai-training with path Allow/Disallow; remember * does not apply to it.
6. Verify after edits: robots.txt syntax, CDN cache of /robots.txt, and Search Console crawl stats — not just a local file save.
7. Content still wins citations: Prefer non-commodity pages and GEO fundamentals over robots-only strategies.

---

FAQ: robots.txt AI training policy token

Is the shared policy token live on Google today?

No. Google announced plans to support it and aims for support with other AI providers in 2027. Until then, Google-Extended remains the company-specific training control to set explicitly.

If I add __usage_ai-training Disallow now, will Google stop training immediately?

Do not assume that. Google’s blog describes future interpretation when the mechanism is supported: opt-out via the policy token with Google-Extended unspecified → treat as Google-Extended disallowed. Honor dates and edge cases will need vendor confirmation at launch.

Does opting out of AI training remove me from AI Overviews?

Not according to Google’s framing of this announcement. Training preferences (policy token / Google-Extended) are separate from Googlebot crawl and from the Search Console generative AI visibility control. Use the right lever for the outcome you want.

Will OpenAI still respect GPTBot rules after the shared signal?

Yes — OpenAI states existing crawler controls remain and continue to apply. The common training signal is an additional way to express foundational generative AI training preferences across participating providers.

Should every site Disallow Google-Extended and the policy token?

Only if refusing Google AI training / Gemini Apps grounding is a deliberate policy. Many publishers keep Google-Extended allowed for Gemini eligibility while still managing other bots and Search generative settings carefully. There is no one “correct” robots.txt for every business model.

---

Bottom line: the Oct 30, 2026 Google + OpenAI announcements are a real step toward a shared robots.txt AI training policy token under IETF AIPREF — but Google’s own support target is 2027, while OpenAI expects to implement in the coming months. Keep Google-Extended and per-bot rules intentional today; treat the new token as a forthcoming cross-vendor layer, not a replacement for Search crawl, generative AI Search controls, or the content work that earns citations.