Scarnox · Blog

Rotating proxies for scraping: setup and pitfalls

A rotating proxy changes the IP address your traffic leaves from, either on every request or after a set interval. It is one of the most common tools in web scraping, and one of the most misused. Rotate too little and a single address gets rate-limited. Rotate too much and the traffic looks nothing like a visitor. This guide explains the two rotation models, shows a working Python setup with retries for dead proxies, and covers how a proxy field fits into captcha solving.

Rotating versus sticky sessions

There are two ways a rotating proxy service hands out addresses:

A real visitor keeps one address for a whole visit. Sticky sessions mirror that behaviour, so they are usually the better default when a site uses cookies or sessions. Use per-request rotation for stateless crawling.

A basic rotating pool in Python

The example below keeps a pool of proxy URLs, rotates through them, and moves on when one fails. Keep credentials in environment variables, never in source code.

import itertools
import os
import requests

# Proxy URLs come from the environment, for example:
# PROXY_POOL="http://user:pass@203.0.113.10:8080,http://user:pass@203.0.113.11:8080"
PROXIES = [p.strip() for p in os.environ.get("PROXY_POOL", "").split(",") if p.strip()]
rotation = itertools.cycle(PROXIES)

def fetch(url: str, max_tries: int = 3) -> requests.Response:
    last_error = None
    for _ in range(max_tries):
        proxy = next(rotation)
        try:
            r = requests.get(
                url,
                proxies={"http": proxy, "https": proxy},
                timeout=30,
            )
            if r.status_code == 407:  # proxy authentication failed
                raise requests.RequestException("proxy auth rejected")
            return r
        except requests.RequestException as exc:
            last_error = exc  # dead or slow proxy: try the next one
    raise RuntimeError(f"all attempts failed for {url}: {last_error}")

Two details matter here. First, a dead proxy should be retried on a different address, not the same one. Second, a request that fails with an authentication error should stop and be reported, rather than burning through the whole pool.

Track proxy health

A plain round robin keeps sending traffic to a proxy that has started failing. Track a simple score for each address: count recent failures, and drop an address from the pool for a cool-down period once it crosses a threshold. Bring it back later and watch it again. Your scraper then spends its requests on the healthy part of the pool.

Log the proxy used with each result. When a site starts returning blocks, you will want to know whether one address, one region or the whole pool is responsible.

Pace the pool

Rotation spreads load across addresses, but each address still has a rate. Set a per-address delay, and a global cap on how many requests you make per second across the whole pool. Rotation is not a licence to send more traffic. It is a way to keep the traffic from one address within reasonable limits.

Common mistakes

Most rotation problems come from a few habits. Rotating the address in the middle of a login or a checkout flow breaks the session and makes the site suspicious. A 429 response is a rate limit on the site, not a broken proxy, so switching to a new address and retrying straight away only spreads the problem. Changing the address on every request while keeping the same cookies is also inconsistent, since the cookies were issued to a different visitor. Finally, a pool that is never checked will quietly fill with dead addresses, and your error rate will climb without an obvious cause. Check the pool on a schedule and remove what fails.

Proxies and captcha solving

Some captcha tasks can use the proxy that your scraper uses. You pass it in the optional proxy field, in the form http://user:pass@host:port, and the solve happens from the same network path your page request uses. Proxyless tasks skip that field entirely. Which one to pick depends on whether the site ties the captcha to the network address, and that trade-off is explained in proxy vs proxyless captcha.

Two practical points:

The live captcha type at Scarnox today is PopularCaptcha. The PopularCaptcha Python guide shows the request and how to submit the returned token.

Choosing your proxies

The type of address you rotate matters as much as the rotation itself. Data-center addresses are fast and cheap per request, but often scored low. Residential addresses score better and are slower. The residential vs datacenter proxies guide compares them in depth, and the advice there applies to the pool you build here.

Scrape responsibly

Next steps

Start with free trial credit, no card needed: create an account and test the PopularCaptcha flow with your own proxy. For current rates, see the pricing page.