SEO

robots.txt

Q: What is the robots.txt?

The robots.txt is a text file in the root directory of a website (/robots.txt) that instructs crawlers which areas they may visit and which they may not. It follows the Robots Exclusion Standard and is respected by Google, Bing and most AI crawlers.

Q: What is the difference between disallow and noindex?

Disallow blocks crawling, noindex prevents indexing. Important: a page blocked by disallow can still end up in the index (without content) if other pages link to it. Anyone wanting to keep a page out of the index permanently uses <meta name="robots" content="noindex"> in the HTML.

Q: Can an incorrect robots.txt ruin SEO?

Yes — immediately and completely. An accidental Disallow: / for user-agent * shuts out all crawlers, and the website disappears from the index within a few days. Classic sources of error: deployment from the staging environment, a CMS plugin update, an accidental wildcard block.

Q: Do I need to address AI crawlers in the robots.txt?

Yes, if visibility in AI answers matters. Without explicit permission, new AI crawlers may be blocked by blanket rules. Best practice: explicitly allow live crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and block training crawlers (GPTBot, ClaudeBot) according to preference.

Q: Where do I place the robots.txt?

In the root of the domain at https://beispiel.de/robots.txt. Subdomains need their own files (blog.beispiel.de/robots.txt). The request must return HTTP 200 — a 404 signals to crawlers that ”everything is allowed”. Rankmio checks robots.txt consistency automatically in the SEO audit.

Also known as: Robots Exclusion Protocol, robots.txt file

The robots.txt is a simple text file in the website root (https://example.de/robots.txt) that tells web crawlers which URLs they may crawl and which they may not. It is based on the Robots Exclusion Protocol (REP), which emerged in 1994 as an informal standard and was formalised as RFC 9309 in 2022. Important: robots.txt prevents crawling, not indexing — and it is binding for reputable crawlers, not for malicious bots.

Structure of a robots.txt

A robots.txt consists of user-agent blocks with allow and disallow rules. Example:

User-agent: *
Disallow: /admin/
Disallow: /search?

User-agent: GPTBot
Disallow: /

Sitemap: https://example.de/sitemap.xml

The first lines block all crawlers from the admin area and internal search result pages. The second block excludes OpenAI's GPTBot entirely. The sitemap line counts as a global directive — even without reference to a user agent.

The most important bot specifications

Googlebot, Bingbot — classic search engine crawlers.
GPTBot (OpenAI), OAI-SearchBot (ChatGPT Search), ClaudeBot, Google-Extended (Gemini training), PerplexityBot — AI crawlers. Blocking them prevents training and answer usage, but also citation chances in AI answers.
AhrefsBot, SemrushBot, MJ12bot — SEO tool crawlers. Blocking them hides the backlink profile from competitors.
* — fallback for all bots not named specifically.

Common robots.txt mistakes

Six classics: (1) Complete block on live (User-agent: * / Disallow: /) after being carried over from staging — the website disappears from the index. (2) Trying to block noindex via robots.txt disallow — this does not work, Google no longer sees the noindex and indexes the URL without a snippet. (3) Blocking JS/CSS prevents correct rendering. (4) Incorrect path patterns — robots.txt is case-sensitive and prefix-based. (5) Forgetting the sitemap directive. (6) A broad disallow wildcard pattern that accidentally blocks money pages as well.

Example from practice

Example: An agency takes over a new client's website and discovers that the robots.txt still contains Disallow: / from the staging phase. Index status according to GSC: 0 indexed pages, site not visible in search results. After the correction and a URL inspection trigger in GSC, 87 % of URLs are indexed again within 10 days — visibility recovers fully after 4 weeks. A classic bug that a pre-launch check would have avoided.

Frequently asked questions

What is the robots.txt?

The robots.txt is a text file in the root directory of a website (/robots.txt) that instructs crawlers which areas they may visit and which they may not. It follows the Robots Exclusion Standard and is respected by Google, Bing and most AI crawlers.

What is the difference between disallow and noindex?

Disallow blocks crawling, noindex prevents indexing. Important: a page blocked by disallow can still end up in the index (without content) if other pages link to it. Anyone wanting to keep a page out of the index permanently uses <meta name="robots" content="noindex"> in the HTML.

Can an incorrect robots.txt ruin SEO?

Yes — immediately and completely. An accidental Disallow: / for user-agent * shuts out all crawlers, and the website disappears from the index within a few days. Classic sources of error: deployment from the staging environment, a CMS plugin update, an accidental wildcard block.

Do I need to address AI crawlers in the robots.txt?

Yes, if visibility in AI answers matters. Without explicit permission, new AI crawlers may be blocked by blanket rules. Best practice: explicitly allow live crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and block training crawlers (GPTBot, ClaudeBot) according to preference.

Where do I place the robots.txt?

In the root of the domain at https://beispiel.de/robots.txt. Subdomains need their own files (blog.beispiel.de/robots.txt). The request must return HTTP 200 — a 404 signals to crawlers that ”everything is allowed”. Rankmio checks robots.txt consistency automatically in the SEO audit.

Used in Rankmio for

robots.txt audit and AI bot configuration

Go to the feature →

Last updated: 2026-06-17 · Browse all glossary entries