Free website tool

robots.txt test — check online in 5 seconds

Is your robots.txt correct? Are Googlebot, Bingbot and AI crawlers like GPTBot or ClaudeBot handled properly? Instant answer, no account.

Part of the full SEO check no account focus: robots.txt

What is checked in detail?

A faulty robots.txt blocks search engines, burns crawl budget or unintentionally excludes your site from AI answers.

What does the result look like?

Sample output with clear status badges — immediately actionable.

Analysis for example.com

✓ OK robots.txt reachable (HTTP 200, 412 bytes)
✓ OK Syntax valid, 4 user-agent blocks found
✓ OK Sitemap reference: https://example.com/sitemap.xml
! Notice GPTBot is blocked — your content will not appear in ChatGPT answers
✗ Error „Disallow: /blog/" likely blocks your entire blog section unintentionally
✓ OK Google-Extended: allowed (Google AI Overviews source possible)

The AI crawlers you should know

They decide whether your site is recommended as an answer source in ChatGPT, Perplexity or Gemini.

GPTBot
OpenAI / ChatGPT — training and live answers
User-agent: GPTBot
ClaudeBot
Anthropic / Claude — answer source
User-agent: ClaudeBot
PerplexityBot
Perplexity AI — AI search engine with source citations
User-agent: PerplexityBot
Google-Extended
Google AI Overviews & Gemini — training crawler
User-agent: Google-Extended

Frequently asked

It tells web crawlers (Googlebot, Bingbot, GPTBot, ClaudeBot etc.) which parts of your site they may visit and which they may not. A misconfigured robots.txt can make entire sections invisible — to Google and to AI answer engines.
Availability (HTTP status, size), syntax validity, every disallow and allow rule per user-agent, sitemap references and whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) are explicitly allowed or blocked.
Zero. No account, no credit card, no newsletter. The test runs directly in the browser using our public crawler infrastructure.
When you want your site to be recommended as an answer source in ChatGPT, Perplexity or Google AI Overviews. GPTBot, ClaudeBot and Google-Extended are the crawlers through which your content lands in AI answers. Blocking them = invisible in AI search.
With an account you get the Robots monitor: daily checks of your robots.txt with email notifications on change, versioning (you see what changed when) and a direct link to your sitemap crawls. Plus the diagnostic "why is page XYZ not being crawled".

Want to monitor robots.txt permanently?

With the free Rankmio account you get daily checks, email alerts on changes and version comparison.

Sign up free →

More free Rankmio tools

Every tool runs without an account. Combine them for a quick technical SEO check or use one in isolation.