AI Crawlers: The Three You're Probably Blocking Wrong
Blocking "AI bots" is a single decision most teams make once, usually in a hurry. It isn't a single decision. The two largest AI companies each run three separate crawlers doing three different jobs, controlled independently, and blocking the wrong one removes you from answers while doing nothing about training.
This covers which crawlers exist, what each does, and what each vendor actually documents, with the primary sources, because this is a page people will edit their robots.txt from.
At a Glance
- Three categories: training collection, search indexing, and user-initiated fetches. Different purposes, different consequences from blocking.
- OpenAI runs GPTBot, OAI-SearchBot and ChatGPT-User. Anthropic runs ClaudeBot, Claude-User and Claude-SearchBot. Each is a separate robots.txt directive.
- Blocking one does not block the others. This is the single most common misconfiguration.
- Blocking training crawlers doesn't remove you from live citations. Blocking search or retrieval crawlers does.
- Most declared crawlers don't execute JavaScript. Content requiring hydration is effectively invisible.
- Some widely discussed crawlers publish no documentation at all, which is itself useful information.
The three categories
Training collection. Gathers content that may be used to train future models. Blocking removes you from future training data. It doesn't affect whether you're cited today, because today's citations come from live retrieval.
Search indexing. Builds the index a system searches when answering. Blocking removes you from those answers.
User-initiated fetching. Retrieves a page because someone in a conversation asked about it. Blocking prevents the assistant completing a task on your behalf.
That split is the whole reason precision matters. A blanket block achieves a training opt-out you may not have needed while closing the retrieval route you wanted.
OpenAI
OpenAI documents three crawlers, each controlled independently:
| Crawler | Purpose | Blocking it means |
|---|---|---|
| GPTBot | Training data collection | Content excluded from future model training |
| OAI-SearchBot | ChatGPT search results | You don't appear in ChatGPT search |
| ChatGPT-User | Fetches during a user's conversation | Assistant can't retrieve your page on request |
Their documentation gives the combination directly: allow OAI-SearchBot to appear in search results while disallowing GPTBot to stay out of training. Robots.txt changes take roughly 24 hours to affect search results.
One caveat worth knowing. OpenAI's documentation has been updated to note that robots.txt may not apply to ChatGPT-User, because a user initiates the fetch rather than a crawler scheduling it. That puts it closer to a browser than to a bot, and it means you have less control over that specific route than the table suggests.
Anthropic
Anthropic restructured its documentation in February 2026 into a matching three-crawler model:
| Crawler | Purpose | Blocking it means |
|---|---|---|
| ClaudeBot | Training data collection | Future materials excluded from training datasets |
| Claude-SearchBot | Search indexing | Reduced visibility and accuracy in Claude search results |
| Claude-User | User-initiated fetches | Claude can't retrieve your content on a user's request |
Three things their documentation states that are worth knowing.
They support Crawl-delay. It's a non-standard extension and not every vendor honours it, but Anthropic explicitly does. Crawl-delay: 1 under a Claude user-agent works.
IP blocking doesn't reliably work. Their words: blocking IP addresses "may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file." Blocking the mechanism that reads your preferences is self-defeating.
Each bot needs its own directive. Blocking ClaudeBot does not block Claude-SearchBot or Claude-User.
Google-Extended is the control for whether your content is used to improve Gemini and related generative models. The important distinction, and one people get wrong: it is not the control for appearing in AI Overviews. That surface draws on Google's standard Search index, so blocking Google-Extended doesn't remove you from AI Overviews, and blocking Googlebot removes you from Search entirely.
If you want to be in AI Overviews but not in Gemini training, Google-Extended is the lever. If you block Googlebot you've made a much larger decision than you intended.
Everyone else
Coverage gets thinner past the two majors, and the honest position is to say so.
PerplexityBot is Perplexity's declared crawler for indexing. They publish documentation and IP ranges for verification.
CCBot belongs to Common Crawl, a non-profit whose dataset feeds many open-source models. Blocking it removes you from that upstream dataset rather than from any single product.
Bytespider (ByteDance) and xAI's Grok crawler are widely discussed and have no official vendor documentation page. That absence is policy-relevant in itself: a vendor documenting purposes, publishing IP ranges and providing a contact address is accountable in a way an undocumented crawler isn't.
Where a vendor publishes nothing, treat any claim about that crawler's behaviour — including claims you read elsewhere — as unverified.
What to actually do
Read your robots.txt properly. Not "are AI bots blocked" but which specific user-agents, and whether that matches what you intended. Most files were written before these crawlers split into three.
Decide per category, not per vendor. Training, search, user-initiated. Most B2B SaaS companies want to allow search and retrieval while making a considered choice on training.
Render server-side. Most declared crawlers don't execute JavaScript, so content that only appears after hydration may as well not exist.
Verify before you block. User-agent strings are trivially spoofed. Anthropic now publishes IP ranges, OpenAI publishes theirs, and reverse DNS lookup confirms a request came from where it claims. Blocking on user-agent alone stops honest crawlers and nothing else.
Don't block by IP where the vendor advises against it, as Anthropic explicitly does.
Monitoring
Check server logs monthly rather than weekly — this changes on a timescale of vendor announcements, not days.
Look for three things. Which AI user-agents are hitting you and how often. Which pages they request, since that tells you what's being read. And any crawler appearing that you don't recognise, which is either a new entrant or something spoofing one.
A monthly log review takes twenty minutes and it's the only way to know whether your robots.txt is doing what you think.
Where Strivelabs fits
Crawler configuration is a one-off technical decision. What it enables is measurable, and that's the harder part to keep an eye on.
Strivelabs samples ChatGPT, Gemini, Claude, Perplexity and Google AI Overviews against prompt sets you define, so you can see whether a robots.txt change actually affected your presence rather than assuming it did. Three samples per prompt, working standalone.
The honest limitation: it doesn't read your server logs or manage crawler access. That stays a technical job for whoever owns the site.
Find out whether that robots.txt change did anything
Strivelabs samples ChatGPT, Gemini, Claude, Perplexity and Google AI Overviews against prompt sets you define — so a change in crawler access shows up as a change in presence, rather than an assumption.
Frequently Asked Questions
How many AI crawlers are there?
More than most teams realise. OpenAI and Anthropic each run three, Google runs Google-Extended alongside Googlebot, and there are several others including PerplexityBot and CCBot. Some widely discussed crawlers publish no documentation at all.
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot handles training. OAI-SearchBot handles ChatGPT search results, and it's a separate directive. Blocking training doesn't remove you from live citations.
Does blocking Google-Extended remove me from AI Overviews?
No. Google-Extended controls Gemini training. AI Overviews draws on Google's standard Search index, so the control there is Googlebot — and blocking Googlebot removes you from Search entirely.
Do AI crawlers respect robots.txt?
Most declared crawlers do, and Anthropic states all three of theirs do. User-initiated fetchers are the exception worth knowing about: OpenAI documents that robots.txt may not apply to ChatGPT-User because a person triggers the request.
Should I block AI crawlers?
Depends on the category. Blocking training is a defensible position with a real cost to future model knowledge. Blocking search and retrieval crawlers removes you from answers today, which for most B2B SaaS companies is the wrong trade.
How do I verify a crawler is genuine?
Reverse DNS lookup on the requesting IP, or check against published IP ranges where the vendor provides them. User-agent strings can be spoofed by anyone.
Does an llms.txt file control crawlers?
No. It's a proposed convention for pointing systems at your key content, not a directive, and no engine has confirmed it as an access control. Robots.txt remains the mechanism.
Related Posts
SaaS SEO for Small Teams: What to Do, In What Order
Most SaaS SEO advice assumes a team you don't have. The sequence for one person: what to fix first, what to build next, and what to deliberately ignore.
What Is Content Engineering? Systems, Not Writing
Duplicate pages, stale posts, slipping refresh cycles. Those aren't writing problems, they're systems problems. What content engineering is and where to start.
Topical Authority: What Google Actually Confirms
Google publishes no topical authority metric, and E-E-A-T isn't a score you optimise for. What's actually observable, and how to build coverage that holds.