---
title: "How to fix HTTP 429 (Too Many Requests) when scraping with AI agents"
category: "guides"
url: https://keirolabs.cloud/blog/fix-429-too-many-requests-scraping-agents
---

A `429 Too Many Requests` is the server telling you to slow down. It is not a crash or a bug in your scraper — it is a rate limit doing its job. You see it most when an AI agent loops over many pages or fires repeated searches, and it shows up fast against search engines, which throttle automated traffic hard. There are three real fixes, in order of effort.

## What a 429 actually means

The HTTP spec (RFC 9110) defines 429 as "the user has sent too many requests in a given amount of time." The response usually carries a **`Retry-After`** header — either a number of seconds or a date — telling you exactly how long to wait. The first rule of handling 429 is: **read `Retry-After` and honor it.** Guessing a delay when the server already told you the answer just causes more 429s.

## Fix 1: exponential backoff with jitter

Instead of a fixed `sleep`, retry with a delay that doubles each time and a random jitter so many clients do not retry in lockstep:

```python
import time, random, requests

def get_with_backoff(url, *, max_attempts=6, base=1.0, cap=60.0, **kw):
    for attempt in range(max_attempts):
        r = requests.get(url, **kw)
        if r.status_code != 429:
            r.raise_for_status()
            return r
        # honor Retry-After if present, else exponential backoff + jitter
        ra = r.headers.get("Retry-After")
        if ra and ra.isdigit():
            delay = float(ra)
        else:
            delay = min(cap, base * (2 ** attempt)) * (0.5 + random.random())
        time.sleep(delay)
    raise RuntimeError(f"still 429 after {max_attempts} attempts: {url}")
```

This goes full speed until the server pushes back, then slows down only as much as it asks. Cap the attempts and the maximum delay so one bad URL cannot hang the whole job.

## Fix 2: rotate proxies / IPs

Backoff helps when the limit is per-connection. When the limit is **per IP**, the only way to send more total requests is to spread them across more source IPs — a rotating proxy pool. Use a residential or datacenter proxy provider, rotate per request or per N requests, and keep the backoff logic on top so a throttled IP still slows down instead of retrying blindly. Rotating a [random user-agent](/blog/random-user-agent) alongside the IP helps a little, but the IP is what rate limits key on.

Proxies add cost and complexity, and for search engines they are a losing battle: Google now requires JavaScript to render results, so even with fresh IPs a plain-HTTP scraper gets thin pages, CAPTCHAs and more 429s.

## Fix 3: stop scraping the SERP — call an API that absorbs the limits

If the thing you are scraping is a **search engine**, the durable fix is to not scrape it yourself. A search API with its own index (or a managed SERP endpoint) has already solved rate limiting, proxying and parsing at their scale, and returns structured results with no 429 for you to manage.

```python
import os, requests

r = requests.post(
    "https://api.keirolabs.cloud/api/v2/search/lite",
    headers={"Authorization": f"Bearer {os.environ['KEIRO_API_KEY']}"},
    json={"query": "latest nextjs 16 middleware api"},
)
results = r.json()  # ranked results, no 429 to retry
```

Keirolabs lite search is **$0.03–$0.10 per 1,000** on its own index; the Google SERP endpoint is a separate, pricier call for when you specifically need Google's results. Either way the rate limits are the provider's problem, not your agent's. [750 free credits, no card](https://platform.keirolabs.cloud/register).

## Which fix should you use?

| Situation | Fix |
| :--- | :--- |
| Occasional 429 from a normal API | Backoff with jitter, honor `Retry-After` |
| Per-IP limit, you own the target | Backoff + rotate proxies |
| Scraping Google/Bing from an agent | Call a search API — don't scrape the SERP |
| High-volume, must not fail mid-task | Search API + backoff on its (generous) limits |

## A note for agent builders

Inside an agent loop, an unhandled 429 usually surfaces as a tool error and derails the run. Wrap every outbound HTTP tool in the backoff helper above, and for web search specifically, prefer a search tool (or [MCP server](/blog/web-search-for-claude-code-codex-mcp)) backed by an API rather than raw scraping. The agent then spends its tokens reasoning, not retrying.