Free website tool

robots.txt test — check online in 5 seconds

Is your robots.txt correct? Are Googlebot, Bingbot and AI crawlers like GPTBot or ClaudeBot handled properly? Instant answer, no account.

Part of the full SEO check no account focus: robots.txt

What is checked in detail?

A faulty robots.txt blocks search engines, burns crawl budget or unintentionally excludes your site from AI answers.

Availability & syntax

HTTP status, file size, encoding and whether the syntax matches the Robots Exclusion Standard. Common bug: typos in directives like „Disalow" instead of „Disallow".

Rules per user-agent

Which disallow and allow rules apply to Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot & co.? Common bug: a global „Disallow: /" that went live by accident.

Sitemap references

Does your robots.txt point to a reachable sitemap? Does the URL resolve? Missing sitemap entries are a standard bug that delays indexing.

AI crawler status

GPTBot, ClaudeBot, PerplexityBot, Google-Extended — are the bots allowed or blocked? Anyone who wants to be cited in ChatGPT, Perplexity or Google AI Overviews must let them through.

Crawl-Delay

Is the non-standard „Crawl-delay" set? Googlebot ignores it, other bots respect it — which can unintentionally slow crawling.

Known anti-patterns

Disallow rules that accidentally block indexed pages (e.g. „/wp-content"), and wildcard patterns that catch more than intended.

What does the result look like?

Sample output with clear status badges — immediately actionable.

Analysis for example.com

✓ OK robots.txt reachable (HTTP 200, 412 bytes)
✓ OK Syntax valid, 4 user-agent blocks found
✓ OK Sitemap reference: https://example.com/sitemap.xml
! Notice GPTBot is blocked — your content will not appear in ChatGPT answers
✗ Error „Disallow: /blog/" likely blocks your entire blog section unintentionally
✓ OK Google-Extended: allowed (Google AI Overviews source possible)

The AI crawlers you should know

They decide whether your site is recommended as an answer source in ChatGPT, Perplexity or Gemini.

GPTBot
OpenAI / ChatGPT — training and live answers
User-agent: GPTBot
ClaudeBot
Anthropic / Claude — answer source
User-agent: ClaudeBot
PerplexityBot
Perplexity AI — AI search engine with source citations
User-agent: PerplexityBot
Google-Extended
Google AI Overviews & Gemini — training crawler
User-agent: Google-Extended

Frequently asked

It tells web crawlers (Googlebot, Bingbot, GPTBot, ClaudeBot etc.) which parts of your site they may visit and which they may not. A misconfigured robots.txt can make entire sections invisible — to Google and to AI answer engines.
Availability (HTTP status, size), syntax validity, every disallow and allow rule per user-agent, sitemap references and whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) are explicitly allowed or blocked.
Zero. No account, no credit card, no newsletter. The test runs directly in the browser using our public crawler infrastructure.
When you want your site to be recommended as an answer source in ChatGPT, Perplexity or Google AI Overviews. GPTBot, ClaudeBot and Google-Extended are the crawlers through which your content lands in AI answers. Blocking them = invisible in AI search.
With an account you get the Robots monitor: daily checks of your robots.txt with email notifications on change, versioning (you see what changed when) and a direct link to your sitemap crawls. Plus the diagnostic "why is page XYZ not being crawled".

Want to monitor robots.txt permanently?

With the free Rankmio account you get daily checks, email alerts on changes and version comparison.

Sign up free →

More free Rankmio tools

Every tool runs without an account. Combine them for a quick technical SEO check or use one in isolation.

XML-Sitemap

Sitemap Tester

Validate XML sitemaps, resolve sitemap indexes recursively, find broken URLs.

Test sitemap → Onpage

Website Crawler

Crawl up to 100 pages: status, title, meta, H1, schema markup, internal links.

Crawl website → PageSpeed

Website Speed Test

Performance, Core Web Vitals, SEO and accessibility score via Google PageSpeed.

Run speed test → robots.txt

robots.txt Check

Fetch file, parse, see all user-agents, allow/disallow rules and sitemaps at once.

Fetch robots.txt → sitemap.xml

Sitemap Check

URL count, sitemap index, lastmod coverage and XML syntax errors in seconds.

Analyze sitemap → JSON-LD

Schema.org Check

Extract JSON-LD, check required fields, detect drift between schema and content.

Check schema → DNS · WHOIS

Domain Check

DNS records (A, AAAA, NS, MX, TXT) plus WHOIS lookup in one overview.

Check domain → Text-Tool

Word Counter

Character, word and sentence count plus reading time. Browser-only, no upload.

Analyze text → Netzwerk

What is my IP?

Your own IP address, browser, language and screen size visible at a glance.

Show IP → Sicherheit

Password Generator

Generate secure passwords in browser — cryptographically strong randomness, no upload.

Generate password →