Scarnox · Blog

Scrape without getting blocked: a practical guide with code

Getting blocked while scraping usually means one of three things: you sent too many requests too fast, your client looked nothing like a browser or a known bot, or your IP address is on a list the site trusts less. You can reduce all three with a few habits: notice the block early, slow down when the site pushes back, keep a consistent and honest identity, and use proxies only where they are appropriate. This guide walks through each with working code. Use it for sites you are allowed to scrape.

Recognise a block

A block does not always look like an error page. Watch for:

Log the status code and a short sample of the body for every failure. A block that you cannot see in your logs is a block you cannot fix.

Back off with retries

When the site says "slow down", your code should slow down by more than a little. Exponential backoff, with a limit on attempts, is the standard approach. It also respects Retry-After when the site sends it:

import time
import requests

session = requests.Session()
session.headers.update({
    "User-Agent": "ExampleBot/1.0 (+https://example.com/bot)",
    "Accept-Language": "en-US,en;q=0.9",
})

def polite_get(url: str, max_attempts: int = 4) -> requests.Response:
    delay = 2.0
    resp = None
    for _ in range(max_attempts):
        resp = session.get(url, timeout=30)
        if resp.status_code not in (429, 503):
            return resp
        retry_after = resp.headers.get("Retry-After", "")
        wait = float(retry_after) if retry_after.isdigit() else delay
        time.sleep(wait)
        delay *= 2
    return resp  # still blocked: return the last response for the caller to handle

resp = polite_get("https://example.com/products?page=1")
print(resp.status_code, len(resp.text))

Note what the function does not do. It does not retry a 403, and it does not rotate through identities to get past one. A 403 is a decision to stop, not a puzzle to solve.

Diagnose a block step by step

When a run starts failing, work through these checks before you change anything else:

  1. Look at the first failing URL and its status code. A block usually starts at a specific point, not at random.
  2. Compare the request rate in the minutes before the first failure. A burst is the most common cause.
  3. Check whether the failure is tied to one path, one section or one time of day.
  4. Fetch the same URL once by hand in a normal browser. If the browser loads it, the difference is in your request pattern. If the browser is blocked too, the site has stopped serving the page to you for now.
  5. Only then change the code. Slow down first, add backoff second, and review your identity last.

Changing several things at once makes it impossible to know which one helped.

Keep a consistent, honest identity

Scrapers get blocked when their identity changes between requests or when it does not match the rest of the traffic. A single session with one clear User-Agent is the simplest stable identity. Name your bot, add a contact link and send only the headers a normal client would send. Changing User-Agents to look like many different visitors is a form of evasion, and it tends to make a block more likely, not less.

If a site serves different content to browsers than to plain HTTP clients, use a real browser for those pages and keep the same rate limit. That is a legitimate route, and it keeps the request pattern sensible.

Use proxies with care

A proxy changes the address a request comes from. It is a tool for legitimate needs, such as testing from a specific region on a site you are allowed to test, or spreading load across your own infrastructure. It is not a way to escape a limit the site has set on you. The format for a proxy is the one used by the proxy task types:

proxies = {
    "http": "http://user:pass@proxy.example.net:8000",
    "https": "http://user:pass@proxy.example.net:8000",
}
resp = session.get("https://example.com/products", proxies=proxies, timeout=30)

Choosing a proxy type is a separate question, covered in residential vs datacenter proxies and rotating proxies for scraping.

When the block becomes a captcha

Some sites answer a rate problem with a captcha rather than a hard block. The sensible order is: stop, slow down, check the rules again, and only then consider a solve. For PopularCaptcha widgets, Scarnox's live API returns a token you submit in the h-captcha-response field. For other captcha types, check the live list, since reCAPTCHA, Cloudflare Turnstile, GeeTest, AWS WAF and Arkose support is coming soon. The detailed guide is web scraping captchas.

If you start seeing blocks on a site, the cause is almost always the request pattern. Avoid captchas when scraping covers the habits that prevent it.

Scrape responsibly

Next steps