FREE-TOOL

robots.txt Check

Fetch, parse and visualize: every user-agent, their allow/disallow rules and all referenced sitemaps. Instant overview, no sign-up.

Enter a domain and the tool fetches /robots.txt automatically over HTTPS. For a deeper check of your own domain — including AI-crawler diagnostics — use the robots.txt test.

More crawler analysis for your own site?

The specialized robots.txt test checks all relevant AI bots (GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, Google-Extended).

robots.txt test →

What robots.txt does — and the AI-crawler reality

robots.txt is a voluntary protocol that tells crawlers which paths they may visit. All major engines respect it; bad-faith scrapers ignore it entirely. For SEO, treating robots.txt as the single source of truth for crawl control is the rule.

The five rules you should know

  • User-agent: — names the crawler the next rules apply to. * matches everyone not explicitly listed.
  • Disallow: /path — blocks that path. Disallow: / blocks the whole site.
  • Allow: /path — overrides a parent Disallow for a sub-path.
  • Sitemap: https://... — points to the XML sitemap. Optional but standard.
  • Crawl-delay: — non-standard. Google ignores it, Bing and Yandex respect it.

The AI-crawler block to know in 2026

If you want to be cited by ChatGPT, Perplexity or Google AI Overviews, you must not block their crawlers. The most common accidental blocks:

  • User-agent: GPTBot + Disallow: / (blocks OpenAI training, but also blocks ChatGPT Search citations).
  • User-agent: Google-Extended + Disallow: / (blocks Gemini and Bard, not regular Google Search).
  • User-agent: anthropic-ai + Disallow: / (legacy Claude crawler).
  • User-agent: ClaudeBot — current Claude crawler used for citations.
  • User-agent: PerplexityBot — required for Perplexity citations.

The drift problem this tool helps with

CMS plugins and CDN edge rules overwrite robots.txt without notice. WordPress with Yoast can append rules a developer didn't write; Cloudflare can serve a different file at the edge than what's on origin. Run this check after every theme update, plugin update or CDN change.

The SEO/GEO angle

A wrong robots.txt is one of the most expensive bugs in SEO — and the easiest to miss. It produces no error in any console, no warning in any audit and no flag in any dashboard. The only signal is a slow, invisible decline in traffic. This tool surfaces it in two seconds.

Frequently asked

robots.txt is a plain-text file at `/robots.txt` that tells cooperative crawlers which URLs they may or may not fetch. It is a request, not a security barrier — well-behaved crawlers (Googlebot, Bingbot, Applebot) respect it, hostile scrapers ignore it. To actually prevent access, use HTTP authentication or IP restrictions. Robots.txt is best understood as „crawl budget management", not „privacy".
Add a `User-agent:` line for each bot followed by `Disallow: /`. As of mid-2026 the important ones are GPTBot (OpenAI training), OAI-SearchBot (ChatGPT search), ClaudeBot (Anthropic), Google-Extended (Google training separate from Search), Meta-ExternalAgent, and Bytespider (TikTok). Blocking training bots keeps your content out of model training; blocking search-user bots removes you from AI-search citations. These are two separate decisions.
It blocks all cooperative bots that read the wildcard `User-agent: *` block. Google, Bing, DuckDuckGo, Applebot, Yandex, Baidu all respect this. But a page can still appear in search results without content (just the URL) if other pages link to it — Google interprets robots.txt as „do not crawl", not „do not index". To prevent indexing entirely, use the `noindex` meta tag on the page itself (which Google can only see if it is allowed to crawl the page).
Yes. Robots.txt supports multiple `User-agent:` blocks. Blocks with a specific bot name apply only to that bot; the `User-agent: *` block applies to everyone not matched by a specific block. Order matters: a bot reads its own block first and ignores the wildcard block entirely, so any wildcard-block rules must be duplicated in the specific block if they should still apply.
When multiple rules could apply to a URL, Google (and most others) pick the rule with the longest matching path — regardless of order in the file. Example: `Disallow: /` plus `Allow: /public/` means everything except `/public/*` is blocked, because `/public/` is longer than `/`. This is how you build sensible allow-list patterns without listing every URL.
It tells crawlers where your sitemap.xml (or sitemap-index) lives. Multiple `Sitemap:` lines are allowed and useful for splitting large sites. Absolute URLs required. This is the fastest way to expose new sitemaps to Bing (which does not have a dedicated „submit sitemap" flow like Google Search Console does).
Google specifies UTF-8. In practice all-ASCII robots.txt files work everywhere; but if you use non-ASCII characters (for example in `#`-comments), they must be UTF-8 encoded without BOM. A BOM at the start of the file can confuse older parsers.

More free Rankmio tools

Every tool runs without an account. Combine them for a quick technical SEO check or use one in isolation.

robots.txt

robots.txt Test

Check crawler rules for Googlebot, GPTBot, ClaudeBot, PerplexityBot in one run.

Check robots.txt → XML-Sitemap

Sitemap Tester

Validate XML sitemaps, resolve sitemap indexes recursively, find broken URLs.

Test sitemap → Onpage

Website Crawler

Crawl up to 100 pages: status, title, meta, H1, schema markup, internal links.

Crawl website → PageSpeed

Website Speed Test

Performance, Core Web Vitals, SEO and accessibility score via Google PageSpeed.

Run speed test → sitemap.xml

Sitemap Check

URL count, sitemap index, lastmod coverage and XML syntax errors in seconds.

Analyze sitemap → JSON-LD

Schema.org Check

Extract JSON-LD, check required fields, detect drift between schema and content.

Check schema → DNS · WHOIS

Domain Check

DNS records (A, AAAA, NS, MX, TXT) plus WHOIS lookup in one overview.

Check domain → Text-Tool

Word Counter

Character, word and sentence count plus reading time. Browser-only, no upload.

Analyze text → Netzwerk

What is my IP?

Your own IP address, browser, language and screen size visible at a glance.

Show IP → Sicherheit

Password Generator

Generate secure passwords in browser — cryptographically strong randomness, no upload.

Generate password →

Free SEO & GEO Check

SEO score, AI visibility and citability of your website in 30 seconds — no registration required.

Check for free now

Ready to optimize your website?

Register for free, get 10 credits and start right away.

Register now