robots.txt Check

What robots.txt does — and the AI-crawler reality

robots.txt is a voluntary protocol that tells crawlers which paths they may visit. All major engines respect it; bad-faith scrapers ignore it entirely. For SEO, treating robots.txt as the single source of truth for crawl control is the rule.

The five rules you should know

• User-agent: — names the crawler the next rules apply to. * matches everyone not explicitly listed.
• Disallow: /path — blocks that path. Disallow: / blocks the whole site.
• Allow: /path — overrides a parent Disallow for a sub-path.
• Sitemap: https://... — points to the XML sitemap. Optional but standard.
• Crawl-delay: — non-standard. Google ignores it, Bing and Yandex respect it.

The AI-crawler block to know in 2026

If you want to be cited by ChatGPT, Perplexity or Google AI Overviews, you must not block their crawlers. The most common accidental blocks:

• User-agent: GPTBot + Disallow: / (blocks OpenAI training, but also blocks ChatGPT Search citations).
• User-agent: Google-Extended + Disallow: / (blocks Gemini and Bard, not regular Google Search).
• User-agent: anthropic-ai + Disallow: / (legacy Claude crawler).
• User-agent: ClaudeBot — current Claude crawler used for citations.
• User-agent: PerplexityBot — required for Perplexity citations.

The drift problem this tool helps with

CMS plugins and CDN edge rules overwrite robots.txt without notice. WordPress with Yoast can append rules a developer didn't write; Cloudflare can serve a different file at the edge than what's on origin. Run this check after every theme update, plugin update or CDN change.

The SEO/GEO angle

A wrong robots.txt is one of the most expensive bugs in SEO — and the easiest to miss. It produces no error in any console, no warning in any audit and no flag in any dashboard. The only signal is a slow, invisible decline in traffic. This tool surfaces it in two seconds.