429 Too Many Requests: Rate Limits and Crawling
A 429 Too Many Requests status means the client hit a rate limit, a signal that shapes how crawlers pace themselves.
429 Too Many Requests is the status code when a client is rate limited.
What It Is
A 429 Too Many Requests means the client exceeded a rate limit.
It protects servers from overload.
Crawlers may back off when they see it.
Why It Matters
- It shields servers from abuse and overload.
- Rate limiting keeps service stable.
- Crawlers respect the signal by slowing down.
- A well-paced crawl beats a throttled one.
- It is part of honest server communication.
How to Do It
- Set sensible rate limits per client.
- Send the Retry-After header with the code.
- Let legitimate bots in at measured rates.
- Log and review 429 patterns.
- Balance crawl access with server health.
What to Avoid
- Rate limiting your own crawler by mistake.
- Missing the Retry-After guidance.
- Blocking all bots as a blunt fix.
- Ignoring 429 spikes from real users.
- Letting limits be so tight they hurt crawl.
Common Mistakes
- Configuring limits that fight your crawl.
- No Retry-After header on 429.
- Rate limiting shared IPs unfairly.
- Forgetting limits after traffic spikes.
- Treating every 429 as an attack.
Before: crawlers flood a server during indexing.
After: rate limits with Retry-After calm the load.
The result: the site stays up and crawling completes steadily.
The lesson: limits are communication, not punishment.
Quick Tip
Pair 429 with a Retry-After header so clients and crawlers know exactly when to try again.
Frequently Asked Questions
429 Too Many Requests, in Short
Too fast is as real as too slow.
Use limits with a clear Retry-After.
Pace the crawl, protect the server.