Google plans to support one robots.txt rule for AI training opt-outs
A shared policy token would set broad AI training preferences alongside company-specific rules, with Google aiming to support it in 2027.
Google plans to support one shared AI training opt-out in robots.txt for participating AI providers. It aims to do that in 2027, so check your file now.
What Google announced
John Mueller announced the plan on the Google Search Central Blog on 30 October 2026. It’s built around a shared ‘policy token’.
The token sits in your robots.txt file and sets broad AI training preferences across participating AI providers. It comes from the IETF’s AIPREF working group.
It’s a draft proposal, Google says.
Say you opt out with the token and Google-Extended is not explicitly specified. Then Google plans to interpret that as if you had disallowed Google-Extended.
Existing controls stay. Company-specific rules like Google-Extended keep working alongside the shared signal, so you can opt out broadly and still tailor rules per company.
Google says other AI providers can handle the token similarly. So a single line in your robots.txt file can speak to more than one company, and you can still pick and choose.
What’s still unclear
So what’s still missing? Four things, and each one changes how you’d use the token:
- What the token’s called and its exact syntax, since the post only calls it a ‘policy token’.
- Which other AI companies will adopt it, and when.
- The date: Google aims for 2027, along with other AI providers.
- Where the rest of the AIPREF vocabulary ends up, since those talks continue.
Why it matters
What should you do this week? Open yoursite.com/robots.txt. Read it line by line.
- Check whether Google-Extended is in your file at all, and whether it’s set to allow or block.
- Note every other AI crawler line, and who added it.
- Decide your stance: opt out entirely, allow everything, or mix the rules by company, depending on what you’d want from Gemini Apps and from other AI providers.
- Save a copy first, so you can roll back.
If you already block Google-Extended, you’ve got a head start. But the shared token only helps if other companies honor it.
Still, RFC 9309 describes robots.txt rules as ones crawlers are requested to honor. It says they’re no form of access authorization, so I’d treat your file as a request.
If you specifically allow Google-Extended, Google says your site is eligible to show up in Gemini Apps responses and get clicks through grounding.
The post doesn’t say what a broad opt-out through the shared token does to that. So weigh it before you block everything.
Will it touch your Search visibility? Treat that as an open question. Watch for Google’s updates.
Google calls Google-Extended its training control, so my guess (and it’s only a guess) is that Search stays separate from it, though I’d want Google to confirm that.
Either way, I like where it’s heading: one opt-out line beats editing a list for every new bot.
Common Questions
When will Google support the shared AI training control in robots.txt?
Google says it aims to support it in 2027, alongside other AI providers. It calls the control a draft proposal and says it will provide updates on its progress.
Does the new robots.txt policy token replace Google-Extended?
No. Google says existing company-specific controls will continue to function alongside the new joint signal. If you opt out with the token and Google-Extended is not explicitly specified, Google plans to interpret that as if you had disallowed Google-Extended.
Which AI companies will support the shared robots.txt policy token?
Google says it aims to support the control along with other AI providers in 2027. It says other providers can handle the token similarly, so site owners can opt out across all providers or pick and choose.
