Crawl Budget: The Number of Pages Search Engines Crawl
Crawl budget is how many pages a search engine crawls on a site within a time frame. Learn what decides it and how to spend it wisely.
Crawl budget is how many pages a search engine crawler is willing to visit on a site within a given time frame.
What Crawl Budget Means
Search engines have a limit on how much they crawl any one site, called the crawl rate, and how many links a crawler will follow once inside, called the crawl demand.
Big sites with millions of pages run out of budget long before the crawler reaches everything. For those sites, what gets crawled is a real strategic choice.
Who Should Care
For small and medium sites, crawl budget is rarely a bottleneck; engines crawl them fully.
It matters for very large sites, sites with many auto-generated URLs, and sites with slow servers that force the engine to back off.
Wasting Your Budget
- Thin pages and duplicate pages the crawler checks again and again.
- Parameterized URLs like product pages with sort filters generating infinite variations.
- Redirect chains and 404s that eat crawl passes.
- Login walls and orphaned content the crawler tries to reach anyway.
Spending It Wisely
- Remove or consolidate thin pages.
- Block low-value parameter URLs in robots.txt.
- Keep internal links pointing at pages that matter.
- List your best pages in the sitemap and update it as content changes.
The site: an e-commerce shop lets filters create millions of URL variations.
The reading: the crawler spends its passes on near-identical filter pages.
The fix: the team blocks the filter URLs in robots.txt and links every product from the category page.
Why it works: the crawler now spends its budget on real products, and they get indexed faster.
Quick Tip
Check the crawl stats report to see pages that returned 404 or that the engine could not finish reading. Each one is budget spent on nothing.
Frequently Asked Questions
Crawl Budget, Bottom Line
Crawl budget is only a real constraint at scale, but when it is, it silently caps how fast you get discovered.
Point every crawl at pages worth keeping, and the budget stops being a problem.