Scarnox · Blog

How to scrape search results past a captcha (Python guide, 2026)

Search result pages are one of the first places a scraper meets a captcha. The site sees a burst of requests from one address, serves a challenge, and returns no results until the challenge is passed. To scrape search results past a captcha, your script needs three steps: detect the challenge before parsing, solve it with one API call, and submit the token so the site returns the real results page.

This guide shows that flow in Python with requests and BeautifulSoup. It covers pages that show a PopularCaptcha widget, which is the type Scarnox solves today (proxyless). Start with free trial credit, no card needed and run the code against a page you own first.

What the challenge looks like

Search pages usually put the widget inside a form. The widget container carries a data-sitekey attribute, and the form posts back to the site:

<form action="/search/verify" method="post">
  <input type="hidden" name="q" value="web scraping">
  <input type="hidden" name="lang" value="en">
  <div class="h-captcha" data-sitekey="YOUR-SITEKEY-UUID"></div>
  <button type="submit">Continue</button>
</form>

Your code needs three things from this markup: the sitekey, the form's action URL, and the hidden fields that must be sent back with the token. The helper below reads all three, so it works without hard-coding them.

If the widget on your target is a different captcha type, see types of captcha. reCAPTCHA and Cloudflare Turnstile support is coming soon, so the code here only covers PopularCaptcha.

The solver call

The solve request goes to the Scarnox API with your key in a Bearer header. The call blocks until the token is ready, so there is no polling loop:

import time
from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup

API_KEY = "scarnox_YOURAPIKEY"
SEARCH_URL = "https://search.example.com/search"

session = requests.Session()
session.headers["User-Agent"] = (
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
    "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36"
)


def solve(sitekey: str, page_url: str) -> str:
    r = requests.post(
        "https://api.scarnox.com/api/tasks/create",
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={
            "captcha_type": "PopularCaptchaTaskProxyless",
            "site_key": sitekey,
            "site_url": page_url,
        },
        timeout=185,  # the call returns when the token is ready
    )
    data = r.json()
    if data.get("errorId") != 0:
        raise RuntimeError(f"{data.get('errorCode')}: {data.get('errorDescription')}")
    return data["solution"]["token"]

A failed solve returns errorId other than 0, and you are not charged for it. Retries are covered in captcha solver error handling.

Fetch the page and pass the captcha

The fetch function checks for the widget first. If there is none, it returns the HTML unchanged. If there is one, it solves the challenge, rebuilds the form with the same hidden fields, and posts it:

def fetch_results(query: str, page: int = 1) -> str:
    resp = session.get(SEARCH_URL, params={"q": query, "page": page}, timeout=30)
    soup = BeautifulSoup(resp.text, "html.parser")

    widget = soup.select_one("[data-sitekey]")
    if widget is None:
        return resp.text  # no challenge on this page

    form = widget.find_parent("form")
    if form is None:
        raise RuntimeError("Captcha widget is not inside a form")

    token = solve(widget["data-sitekey"], resp.url)

    payload = {
        field["name"]: field.get("value", "")
        for field in form.select("input[name]")
    }
    payload["h-captcha-response"] = token
    payload["g-recaptcha-response"] = token

    action = urljoin(resp.url, form.get("action") or resp.url)
    verified = session.post(action, data=payload, timeout=30)

    if "data-sitekey" in verified.text:
        raise RuntimeError("Challenge still shown after submit; token may be rejected")
    return verified.text

Two details matter here. The site_url is the URL the captcha is actually served on (resp.url, which includes any redirects), not the bare search endpoint. And the session keeps cookies between the GET and the POST, which many sites need to accept the token.

Tokens expire in about two minutes, which is why the solve happens right after the form is read, not before. For more on that window, see captcha token expiry.

Parse the results

Once the real results page is returned, parsing is ordinary HTML work. The selectors below are placeholders; open the page in DevTools and replace them with the real ones:

def parse_results(html: str) -> list[dict]:
    soup = BeautifulSoup(html, "html.parser")
    rows = []
    for item in soup.select("div.result"):
        link = item.select_one("a[href]")
        heading = item.select_one("h3")
        if link and heading:
            rows.append({
                "title": heading.get_text(strip=True),
                "url": link["href"],
            })
    return rows


if __name__ == "__main__":
    for page in range(1, 4):
        html = fetch_results("web scraping", page)
        for row in parse_results(html):
            print(row)
        time.sleep(5)  # pace requests between pages

Keep the output of each page separate. If a page returns zero rows, check for the widget before assuming the query had no results.

Scrape responsibly

A captcha is a signal, not a loophole. Before you run this against any search site:

For the legal side, see is web scraping legal. For avoiding blocks in the first place, see scrape without getting blocked.

Pricing

Scarnox charges per successful solve, and failed solves are free. See current pricing for the rates. Start with free trial credit, no card needed.

Next steps