GEO / AI search

AI crawler

Updated

Also known as: LLM crawler, AI bot, GPTBot, ClaudeBot, Google-Extended, PerplexityBot

AI crawlers (also called LLM crawlers or AI bots) are programs run by AI providers that fetch web pages. They do so for two different purposes: to collect text for training future language models, or to read a page for search and current answers, for example in ChatGPT, Claude or Perplexity. Many providers use separate crawlers for this. If you want to be named in AI answers, you have to allow the search crawlers; you can still block the training crawlers.

Author: Knut Nickol | Updated: 21 November 2026

The most important AI crawlers and what they do

CrawlerPurposeEffect of blocking it
GPTBot (OpenAI)TrainingContent is not used for training; ChatGPT search is not affected
OAI-SearchBot (OpenAI)Search in ChatGPTThe page does not appear in ChatGPT search results
ChatGPT-User (OpenAI)Fetch when a user asks for itAccording to OpenAI, robots.txt may not apply here
ClaudeBot (Anthropic)TrainingContent is not used for training
Claude-SearchBot (Anthropic)Search in ClaudeLess visibility in Claude's search results
Claude-User (Anthropic)Fetch when a user asks for itClaude cannot read the page for user questions
PerplexityBot (Perplexity)Search in Perplexity, according to Perplexity not used for trainingThe page does not appear in Perplexity search results
Perplexity-User (Perplexity)Fetch when a user asks for itAccording to Perplexity mostly ineffective, this fetcher generally ignores robots.txt
Google-Extended (Google)Not a crawler of its own but a control token for Gemini and the training of Google modelsNo effect on Google Search and its AI Overviews
Meta-ExternalAgent (Meta)Training and indexing for Meta AIContent is not used by Meta AI
Bytespider (ByteDance)TrainingContent is not used for training
CCBot (Common Crawl)Open web archive that many AI companies use for trainingThe page is missing from future archive snapshots
Amazonbot (Amazon)Alexa and other Amazon servicesContent is missing from Amazon's answers

The AI Overviews in Google Search are built from the normal Google index. If you do not want to appear there, the only lever is the rules for Googlebot, and then the page also disappears from normal search.

The user agent alone does not prove that a real AI crawler is calling

The user agent is freely chosen text. In an analysis of the rankmio.de server logs over 14 days with 171,590 requests (October 2026), attack attempts arrived under the names of real AI bots, for example an attempt to read system files using the ClaudeBot identifier. Three addresses from Google Cloud switched every minute between GPTBot, ChatGPT-User, OAI-SearchBot, Claude-User and PerplexityBot.

You recognise real crawlers by their origin (IP address), not by their identifier. The large providers publish the address ranges of their crawlers, OpenAI and Perplexity for example as downloadable lists. Anyone counting AI crawler visits should therefore check both, otherwise attackers count towards AI visibility.

AI crawling strategy: which to allow, which to block

  1. Allow everything — the best chance of being named in AI answers. Your content may then also be used for training.
  2. Search yes, training no — allow search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot), block training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot). The page stays visible in AI answers without agreeing to training. The sensible middle ground for most businesses.
  3. Block everything — largely protects content from AI use. But the page then disappears from ChatGPT, Claude and Perplexity; it can still appear in Google's AI Overviews as long as Googlebot may read it.

A concrete robots.txt example (variant 2)

# Block training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

# Allow search crawlers: no entry = allowed
# OAI-SearchBot, Claude-SearchBot, PerplexityBot are not listed

Example from practice

How to check your site: rankmio's free robots.txt test reads your robots.txt and shows separately which search crawlers (Googlebot, Bingbot, OAI-SearchBot, PerplexityBot, Claude-SearchBot) and which training crawlers (GPTBot, ClaudeBot, Google-Extended) may fetch your home page. The most common finding: a block on "all AI bots" from a plugin or template also hits the search crawlers, and the page is missing from AI answers without anyone noticing.

Frequently asked questions

What is an AI crawler?
An AI crawler is a program from an AI provider (OpenAI, Anthropic, Google, Perplexity) that fetches web pages: either to collect text for training future language models or to read the page for search and current answers. Examples: GPTBot (training at OpenAI), OAI-SearchBot (search in ChatGPT), ClaudeBot (training at Anthropic), PerplexityBot (search in Perplexity).
Should you block AI crawlers?
It depends on your goal. If you want to be named in AI answers, you have to allow the search crawlers, otherwise ChatGPT, Claude or Perplexity will not find the page. Training crawlers can be blocked separately if content should not feed into future models. A complete block costs your visibility in these AI services.
How do you block an AI crawler?
Via robots.txt, with a separate entry for each crawler:
User-agent: GPTBot
Disallow: /

Fetches triggered by a user (ChatGPT-User, Perplexity-User) do not always follow robots.txt, according to the providers. To shut them out reliably, you have to do it on the server, for example using the published IP addresses.
Do AI crawlers obey robots.txt?
The crawlers of the large providers do: OpenAI, Anthropic, Google and Perplexity document their crawlers and follow robots.txt. Exceptions are fetches a user triggers directly, and individual scrapers posing as well-known crawlers. For sensitive content, only a check on the server helps.
Which AI crawlers exist in 2026?
More than 20 known identifiers. The most important: GPTBot, OAI-SearchBot, ChatGPT-User (OpenAI), ClaudeBot, Claude-SearchBot, Claude-User (Anthropic), Google-Extended (Google), PerplexityBot, Perplexity-User (Perplexity), Meta-ExternalAgent (Meta), Bytespider (ByteDance), CCBot (Common Crawl), Amazonbot (Amazon). New crawlers are added every few months.
Sources

Used in rankmio for

AI bot configuration in the robots.txt audit

Go to the feature →

Last updated: 2026-11-21  ·  Browse all glossary entries

Free SEO & GEO Check

SEO score, AI visibility and citation readiness of your website in 30 seconds — no registration required.

Check for free now

Ready to optimize your website?

Register for free, get 10 credits and start right away.

Register now