Vercel Lets Sites Control AI Crawlers and Assistants Separately

Vercel’s Firewall now has separate controls for AI crawlers and AI assistants, letting site owners choose which kinds of AI bots reach their pages (Vercel’s announcement).

The ruleset is available on all plans.

Crawlers collect content for indexing and model training, while assistants fetch pages after a user action or search query, so you can now treat the two very differently on your site.

Choose what each bot can do

Vercel says you can block crawlers while still allowing assistants to retrieve content for answers. Requests from either category can also be logged.

The company says it updates the rules as it adds bots.

Want to limit collection but keep your pages available when someone asks an assistant a question? That’s the choice these controls offer.

Where to change the rules

In your Vercel project, go to Security > Firewall > Rules. Turn on AI Bots in the Bot Management section and set a separate action for crawlers and for assistants before you publish.

There’s a CLI option too. Vercel’s changelog shows this command for denying crawlers:

vercel firewall rules edit ai-bots --category ai-crawlers --action deny --yes

Then review the proposed changes with vercel firewall diff and apply them with vercel firewall publish.

Why it matters

This affects any site owner who wants a say in how AI services access their content, since you could block collection bots and still let an assistant fetch an article when a reader asks.

Still, the right setting depends on what you want your site to do. Want assistants to send readers to your pages? Blocking every AI bot could close that route.

Concerned about collection? Separate categories let you act without automatically blocking assistants too.

Vercel’s announcement doesn’t name the bots covered, so review your project’s logs and current documentation before relying on a specific bot being included.

So log first.

You can log requests from each category before you block anything, and then you can decide whether each type fits your content and audience.

The control itself is useful.

I’d treat it as an access decision and watch your own traffic.

Common Questions

Can I block AI crawlers but allow AI assistants in Vercel?

Yes. Vercel’s AI Bots managed ruleset lets you set separate actions for AI crawlers and AI assistants, including blocking crawlers while allowing assistant requests.

Is Vercel’s AI Bots Firewall ruleset available on every plan?

Yes. Vercel says the managed AI Bots ruleset is available on all plans.

How do I change AI bot rules in Vercel Firewall?

Open your project’s Security > Firewall > Rules, turn on AI Bots in the Bot Management section, and set a separate action for crawlers and for assistants. Then review and publish the changes.