Scarnox · Blog

How to scrape a Cloudflare-protected site (2026 guide)

Cloudflare sits in front of a large share of the public web, so a scraper that works on one site will often hit a wall on the next. You usually notice it as an HTTP 403 or 503, a "Just a moment..." interstitial, or a Turnstile checkbox on the page. This guide explains what Cloudflare is checking, the fixes that work without touching anything you should not touch, and where captcha solving fits. The short answer: confirm you may scrape the site, slow down, send requests that look like a normal client, use a real browser when the page needs JavaScript, and only then deal with a captcha. Turnstile is the captcha Cloudflare uses most often. Scarnox support for Turnstile is coming soon. The captcha type that is live today is PopularCaptcha.

What Cloudflare is checking

Cloudflare scores every request on several signals:

IP reputation. Addresses from cloud data centers start with a lower score than addresses from home connections.
Request rate and pattern. Hundreds of requests per second from one address look like a bot, whatever the headers say.
Headers and TLS fingerprint. A bare HTTP library often sends headers, header order and a TLS handshake that no mainstream browser sends.
JavaScript execution. The challenge page runs scripts and checks that a real browser environment is present.
Cookies. After a challenge is passed, Cloudflare issues a clearance cookie tied to that session.

A high score passes silently. A middling score gets an interactive challenge. A low score gets a block. Most of the work is about moving your score up by behaving like a well-behaved client, not by fighting the check.

Step 1: check that you should scrape the site

Before any technical work, read the site's robots.txt and its terms of service. A block is often the owner telling you something. Scraping public pages slowly and for a clear purpose is one thing. Forcing past a block to reach data the owner has restricted is another, and it is a use Scarnox does not support. The legal overview covers the general questions.

Step 2: make the plain request right

Many scrapers that look "blocked" are simply sending poor requests. Fix these first:

Send a current User-Agent and the headers a browser would send, such as Accept and Accept-Language.
Keep one session object so cookies persist between requests, as a browser would.
Add a delay between requests, with a little random jitter.
Back off on 429 and 503 responses instead of retrying in a tight loop.

import random
import time
import requests

session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/124.0 Safari/537.36",
    "Accept-Language": "en-US,en;q=0.9",
})

def polite_get(url: str, attempts: int = 4) -> requests.Response:
    r = None
    for attempt in range(attempts):
        r = session.get(url, timeout=30)
        if r.status_code not in (429, 503):
            return r
        # Exponential backoff with jitter: about 1, 2, 4 seconds plus noise.
        time.sleep(2 ** attempt + random.uniform(0, 1))
    return r

If this gets you through, you are done. Keep the delay in place.

Step 3: work out which challenge you are facing

The response tells you a lot:

A "Just a moment..." page. This is a JavaScript challenge. A real browser usually passes it on its own after a few seconds. The clearance cookie can then be reused for later requests in the same session.
A Turnstile widget on a form or page. This is a captcha. It needs a token from the widget before the form will accept a submission.
A block page with an error code. The address or the request has been refused outright. Changing your pacing and fixing your requests usually helps more than anything else. Solving does not turn a hard block into data.

Checking which of these you see saves a lot of wasted effort.

Step 4: use a real browser when the page needs it

If the page only shows content after JavaScript runs, a plain HTTP client will never see it. Drive a full browser with Playwright or Selenium instead. A headless browser is detectable, so read headless browser detection before you rely on one, and set it to match the environment a real visitor would have. The JavaScript site guide shows a complete example.

Step 5: look at your IP reputation

Cloud data-center addresses are the most common reason a clean request gets a low score. Residential addresses usually score better, at a higher cost in speed and money. The trade-off is covered in residential vs datacenter proxies.

Rotation has its own traps. Changing the address on every request looks unnatural, because a real visitor keeps one address for a whole visit. Sticky sessions, where one address is kept for a set of related requests, often do better than rapid rotation. Rotating proxies for scraping goes into the details.

Step 6: where captcha solving fits

Sometimes the widget does appear, on a form you are testing or on a page you have permission to automate. A Turnstile token is then a required step in that flow. Scarnox support for Turnstile is coming soon, so do not build production code against it yet. The live captcha type today is PopularCaptcha, which uses a different widget.

The general workflow for any token-based captcha is the same. You read the sitekey from the page, request a token, and place the token in the form field the site expects. Tokens are short-lived, so submit them straight away. The PopularCaptcha Python guide shows that workflow end to end, and captcha types explained compares the widgets you will meet.

If you want to be ready when Turnstile support goes live, you can start a free trial now. Start with free trial credit, no card needed: create an account. For the current rates of the live captcha type, see the pricing page.

Scrape responsibly

Read robots.txt and honour the terms of the site you scrape.
Rate-limit your crawler, and run it at off-peak hours where you can.
Cache responses so you never fetch the same page twice for no reason.
Avoid collecting personal data unless you have a lawful basis for it.
If a site has restricted access, stop. Do not try to get past the restriction.