Search

Search PublishMyBlog

SEO

Google's Gary Illyes Blames AI for More Fake Googlebots

A user-agent that says Googlebot proves nothing; a DNS lookup does, and most small sites should look before they block.

By , Editor

4 min read

A hand holds a Google ID badge at a lobby security desk, beside a magnifying glass and a badge reader with a red light

Bots that pretend to be Googlebot are a bigger problem now, and Google’s Gary Illyes blames AI. Do you wave through any bot that claims to be Googlebot? Some of those visits aren’t Google.

Illyes talked bots with Aneta Samkoff on the Odys Podcast, as Search Engine Roundtable reported on October 30.

Asked why impersonation has grown, he said: “I think it is AI.”

Why fakes say they’re Googlebot

More and more companies want content to train their own models, and Illyes describes the web as a big farm they think they can just go out and harvest.

Firewalls and CDNs catch more crawlers than they used to, but not all of them.

So a small AI engine or a scraper can claim to be Googlebot and hope you won’t block it. Simple.

Illyes added three things you should know about the fakes:

  • Anyone can claim to be Googlebot in seconds, because the user-agent is just a header, and his honeypots caught “Googlebots” from IPs unrelated to Google.
  • A badly built bot can hammer your URLs until real visitors are locked out. He puts much of that down to ignorance or poor setup.
  • Your robots.txt is only a suggestion, he says, so a bot that wants your pages can ignore it.

What Illyes sees coming

He raised three ideas, at very different stages:

  • Finer controls per type of bot, like Google-Extended and Apple’s Applebot-Extended today.
  • Web Bot Auth: signed requests your site checks against a public key.
  • A crawler reputation score built from site and security company reports, which he calls early and says could carry bias if one company runs it.

I’d put Google’s shared AI training opt-out in robots.txt in the same family. Still, only signed requests would catch a fake Googlebot that lies about who it is.

How to check a Googlebot is real

Google’s guide to verifying its crawlers gives you two ways to check a Googlebot is real. For a one-off check, take an IP from your server logs and run two lookups:

host 66.249.66.1
host crawl-66-249-66-1.googlebot.com

Then check two things:

  1. The first lookup gives a name ending in googlebot.com, google.com or googleusercontent.com.
  2. The second returns the IP you started with.

Fail either, and your “Googlebot” is someone else.

Got lots of traffic to check?

Then match your server logs against the IP ranges Google publishes as JSON files, like common-crawlers.json, and flag every bot that claims to be Googlebot from an IP outside them.

One catch from the same doc: user-triggered fetchers ignore robots.txt.

Why it matters

My read: before you have a fake Googlebot problem, you probably have a server logs problem, because nobody on your team has opened them since the site went live.

Illyes said something close. His advice for small sites is to check whether bots really hurt before you pay for protection, and I’d budget an hour for it. Here’s how I’d do it:

  1. Filter your server logs, or your host’s analytics, for any bot that claims to be Googlebot. Seeing a spike?
  2. Compare the volume with the crawl stats in Search Console.
  3. Run the DNS check on your busiest IPs, and keep the ones that pass.

If they’re fake and slowing your site, your next step is a WAF or a CDN with bot management. On Shopify and similar hosted platforms, you already have some protection.

Meanwhile, on Vercel you get separate controls for AI crawlers and assistants.

That said, never block by user-agent alone. Any scraper can type that header.

Check the IP.

Common Questions

How do I know if a visitor claiming to be Googlebot is really from Google?

Run a reverse DNS lookup on the IP from your server logs, check the hostname ends in googlebot.com, google.com or googleusercontent.com, then do a forward lookup and confirm it returns the IP you started with.

Does robots.txt block fake Googlebots and AI scrapers?

No. Gary Illyes calls robots.txt a suggestion that compliant crawlers follow and others can ignore. Blocking a bot that ignores it needs a web application firewall or a CDN with bot management.

Why are more bots pretending to be Googlebot in 2026?

Google’s Gary Illyes says AI is the reason: companies want web content to train models, and a scraper or small AI engine that claims to be Googlebot is less likely to be blocked by a site.

All SEO news