AI crawler
Updated
Also known as: LLM crawler, AI bot, GPTBot, ClaudeBot, Google-Extended, PerplexityBot
AI crawlers (also called LLM crawlers or AI bots) are programs run by AI providers that fetch web pages. They do so for two different purposes: to collect text for training future language models, or to read a page for search and current answers, for example in ChatGPT, Claude or Perplexity. Many providers use separate crawlers for this. If you want to be named in AI answers, you have to allow the search crawlers; you can still block the training crawlers.
Author: Knut Nickol | Updated: 21 November 2026
The most important AI crawlers and what they do
| Crawler | Purpose | Effect of blocking it |
|---|---|---|
| GPTBot (OpenAI) | Training | Content is not used for training; ChatGPT search is not affected |
| OAI-SearchBot (OpenAI) | Search in ChatGPT | The page does not appear in ChatGPT search results |
| ChatGPT-User (OpenAI) | Fetch when a user asks for it | According to OpenAI, robots.txt may not apply here |
| ClaudeBot (Anthropic) | Training | Content is not used for training |
| Claude-SearchBot (Anthropic) | Search in Claude | Less visibility in Claude's search results |
| Claude-User (Anthropic) | Fetch when a user asks for it | Claude cannot read the page for user questions |
| PerplexityBot (Perplexity) | Search in Perplexity, according to Perplexity not used for training | The page does not appear in Perplexity search results |
| Perplexity-User (Perplexity) | Fetch when a user asks for it | According to Perplexity mostly ineffective, this fetcher generally ignores robots.txt |
| Google-Extended (Google) | Not a crawler of its own but a control token for Gemini and the training of Google models | No effect on Google Search and its AI Overviews |
| Meta-ExternalAgent (Meta) | Training and indexing for Meta AI | Content is not used by Meta AI |
| Bytespider (ByteDance) | Training | Content is not used for training |
| CCBot (Common Crawl) | Open web archive that many AI companies use for training | The page is missing from future archive snapshots |
| Amazonbot (Amazon) | Alexa and other Amazon services | Content is missing from Amazon's answers |
The AI Overviews in Google Search are built from the normal Google index. If you do not want to appear there, the only lever is the rules for Googlebot, and then the page also disappears from normal search.
The user agent alone does not prove that a real AI crawler is calling
The user agent is freely chosen text. In an analysis of the rankmio.de server logs over 14 days with 171,590 requests (October 2026), attack attempts arrived under the names of real AI bots, for example an attempt to read system files using the ClaudeBot identifier. Three addresses from Google Cloud switched every minute between GPTBot, ChatGPT-User, OAI-SearchBot, Claude-User and PerplexityBot.
You recognise real crawlers by their origin (IP address), not by their identifier. The large providers publish the address ranges of their crawlers, OpenAI and Perplexity for example as downloadable lists. Anyone counting AI crawler visits should therefore check both, otherwise attackers count towards AI visibility.
AI crawling strategy: which to allow, which to block
- Allow everything — the best chance of being named in AI answers. Your content may then also be used for training.
- Search yes, training no — allow search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot), block training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot). The page stays visible in AI answers without agreeing to training. The sensible middle ground for most businesses.
- Block everything — largely protects content from AI use. But the page then disappears from ChatGPT, Claude and Perplexity; it can still appear in Google's AI Overviews as long as Googlebot may read it.
A concrete robots.txt example (variant 2)
# Block training crawlers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
# Allow search crawlers: no entry = allowed
# OAI-SearchBot, Claude-SearchBot, PerplexityBot are not listed
Example from practice
How to check your site: rankmio's free robots.txt test reads your robots.txt and shows separately which search crawlers (Googlebot, Bingbot, OAI-SearchBot, PerplexityBot, Claude-SearchBot) and which training crawlers (GPTBot, ClaudeBot, Google-Extended) may fetch your home page. The most common finding: a block on "all AI bots" from a plugin or template also hits the search crawlers, and the page is missing from AI answers without anyone noticing.
Frequently asked questions
What is an AI crawler?
Should you block AI crawlers?
How do you block an AI crawler?
User-agent: GPTBot
Disallow: /Fetches triggered by a user (ChatGPT-User, Perplexity-User) do not always follow robots.txt, according to the providers. To shut them out reliably, you have to do it on the server, for example using the published IP addresses.
Do AI crawlers obey robots.txt?
Which AI crawlers exist in 2026?
Used in rankmio for
AI bot configuration in the robots.txt audit
Free SEO & GEO Check
SEO score, AI visibility and citation readiness of your website in 30 seconds — no registration required.
Ready to optimize your website?
Register for free, get 10 credits and start right away.
Register now