Crawling
Updated
Also known as: Web crawling, Crawl, Spidering
Crawling is the process in which search engine bots (Googlebot, Bingbot and so on) automatically retrieve web pages, parse their HTML, extract new links and place those links in a queue for further crawling. Crawling is the first stage in the pipeline: crawl → render → index → rank. Anything that is not crawled does not appear in the index and cannot rank. On large websites, crawl budget — the crawl capacity Google assigns to a domain — is a limiting factor.
Understanding crawl budget
Google allocates each domain a crawl budget, which is made up of two components: the crawl rate limit (how many requests per second can the server handle?) and crawl demand (how important does Google consider the domain and its content?). On small sites (< 10,000 URLs) budget is not an issue — Google crawls everything. From around 100,000 URLs onwards budget becomes relevant: not all URLs are crawled, and some only every few weeks. On very large sites (millions of URLs) budget management is essential.
What consumes crawl budget
- Endless filter and parameter variations in shop categories.
- Soft 404s (status 200, but "page not found" content).
- Long redirect chains.
- Poor server response times (Google throttles crawling).
- Indexable duplicate pages without a canonical.
- Crawlable login and search result pages that contribute nothing.
Controlling crawling
- robots.txt — exclude entire URL groups from crawling.
- Canonical tags — signal duplicates and focus budget on the canonical URL.
- Sitemap.xml — communicate priorities clearly.
- Internal linking — link to important URLs prominently and hold back peripheral ones.
- Last-Modified header — a server response with a genuine modification date means Google crawls unchanged pages less often.
- Server performance — fast responses mean a higher crawl limit.
- Search Console settings — throttle the crawl rate manually (rarely necessary).
Example from practice
Example: A shop with 38,000 URLs had the following crawl statistics in GSC: Googlebot crawled 22,000 URLs a day, 60% of which were filter and sorting parameter variants. Setting canonical tags on all parameter URLs, plus Disallow: /search? and targeted wildcard blocks in robots.txt, reduced the crawl to 8,500 URLs a day, all of them canonical products. New products now appear in the index after an average of 4 days instead of 14 — with the same crawl effort from the bot.
Frequently asked questions
What is crawling in SEO?
What is a crawl budget?
How can I see crawl behaviour?
How do you control crawling?
Used in rankmio for
Crawl findings in the site audit and crawl statistics from Bing
Free SEO & GEO Check
SEO score, AI visibility and citation readiness of your website in 30 seconds — no registration required.
Ready to optimize your website?
Register for free, get 10 credits and start right away.
Register now