Scarnox · Blog

Web scraping captchas: why they appear and what to do

A captcha shows up during scraping when a site decides a request looks automated. The usual triggers are too many requests in a short time, an IP address with a poor reputation, missing browser signals, or a hit on a page the site protects more closely than the rest. The practical response has three steps, in this order: reduce the triggers, detect the captcha in code, and solve the captcha you cannot avoid. This guide covers each step and points to the detailed guides for the rest.

Why sites show captchas to scrapers

A site has little way to tell a person from a script at the moment a page is requested, so it looks at patterns. A scraper often stands out because it:

sends requests faster than any person would click,
repeats the same path with no pause,
arrives from a datacenter address rather than a home network,
sends headers that do not match a common browser, or
keeps no cookies or session state between requests.

Any one of these may be enough. The captcha is the site's way of asking for a human-like step before it serves more pages. Reducing the pattern usually reduces the captchas, which is cheaper than solving them.

Detect the captcha in code

Before you solve anything, check whether the page you got is the captcha page at all. A simple test looks for the widget markup and reads the sitekey:

import re
import requests

SITEKEY_RE = re.compile(r'data-sitekey="([0-9a-f-]{36})"')

def fetch(url: str) -> dict:
    resp = requests.get(
        url,
        timeout=30,
        headers={"User-Agent": "ExampleBot/1.0 (+https://example.com/bot)"},
    )
    html = resp.text
    if "h-captcha" in html or "g-recaptcha" in html:
        match = SITEKEY_RE.search(html)
        return {"captcha": True, "sitekey": match.group(1) if match else None, "html": html}
    return {"captcha": False, "sitekey": None, "html": html}

page = fetch("https://example.com/search?q=shoes")
if page["captcha"]:
    print("captcha found, sitekey:", page["sitekey"])

The h-captcha and g-recaptcha strings are only the markup a page uses, so this is a signal rather than a guarantee. If the sitekey is missing, see how to find a captcha sitekey.

Solve it when you cannot avoid it

When a PopularCaptcha widget is on the page and you have its sitekey, you can request a token from the Scarnox API with one POST request and submit it in the h-captcha-response field. The full function, with error handling, is in solve a captcha from Python with one request. Use the token right away, because it expires after about two minutes. The timing details are in captcha token expired.

Scarnox's live support today is PopularCaptcha, proxyless and with your own proxy. reCAPTCHA v2 and v3, Cloudflare Turnstile, GeeTest, AWS WAF and Arkose are coming soon. If your target uses one of those, the captcha guides explain the general options, but do not build a production flow on a solve that is not live.

A decision flow for each blocked page

When your scraper hits a captcha, a simple decision order keeps the job honest and keeps the request volume down:

Is there a data source you can use instead? If an API, feed or export covers the fields you need, switch to it and stop requesting that page.
**Does robots.txt allow the path?** If not, skip it. Do not solve a captcha on a page you were asked to leave alone.
Did you send too much too fast? Pause for a longer period, lower your rate and check again later. Many captchas clear up with a slower pace.
Is the captcha a live type? If it is PopularCaptcha, the solve in this guide applies. If it is a type that is coming soon, keep the job paused and monitor for the release.
Is the page one you are allowed to automate? If the answer is unclear, ask the site owner before you continue.

The fourth step is the one most often skipped. A solver is only useful when the captcha type is supported, and a job that retries an unsupported type wastes time and requests.

Handle the captcha in a job, not in a loop

Put the solve inside the code path that needs the token, so the job can log each captcha, stop when a page keeps failing and move on. An unbounded loop that keeps requesting a captcha page is the fastest way to get a site's firewall to block your address for good.

Monitor what fails

Keep a count of pages that return a captcha, grouped by URL and by time of day. A captcha that appears only on one section of a site, or only after a certain number of requests, tells you where the trigger is. That information is more useful than a faster solver, because it shows you what to change.

What not to do

Do not rotate through many IP addresses to avoid a site's limits. That is evasion, and it turns a slow-down request into a fight.
Do not scrape pages behind a login with credentials you are not authorised to use.
Do not solve captchas to create accounts, post spam or buy limited items.

Scrape responsibly

Read the site's robots.txt and terms of service before you start. Respect the disallowed paths and any crawl delay.
Identify your bot honestly in the User-Agent, with a way to contact you.
Keep the request rate low. A captcha is often the site asking you to slow down.
Stop if the site asks you to, and do not work around a block you were told to respect.
Handle personal data under the privacy rules that apply to you, and collect only what you need.