Scarnox · Blog

Scrapy captcha middleware: solve captchas in your spider (2026)

Scrapy is built around requests and callbacks, so a captcha does not fit neatly into it. The clean way to handle one is a downloader middleware: it sees every response, notices a challenge, gets a token, and lets the spider submit the form with that token. This guide builds that middleware for PopularCaptcha pages, which Scarnox solves today on a proxyless basis.

Start with free trial credit, no card needed and try the code against a test page before you point it at anything else.

How the middleware works

The middleware runs on every response:

  1. If the response has no data-sitekey, it passes the response through.
  2. If it does, the middleware solves the captcha and returns a copy of the same request with the token stored in meta.
  3. Scrapy fetches that request again. The middleware sees the token in meta and lets the response through.
  4. The spider reads the token from meta and submits the form with FormRequest.from_response.

The solve call runs in a thread through deferToThread, so it does not block Scrapy's reactor. Other requests keep moving while the captcha is being solved.

Project settings

Put your key in an environment variable, not in the code:

# settings.py
import os

SCARNOX_API_KEY = os.environ["SCARNOX_API_KEY"]
SCARNOX_TIMEOUT = 185

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.PopularCaptchaMiddleware": 543,
}

ROBOTSTXT_OBEY = True
DOWNLOAD_DELAY = 3
CONCURRENT_REQUESTS_PER_DOMAIN = 1

The download delay and single concurrency per domain are there on purpose. They keep the site's load low and reduce the number of challenges you see.

The middleware

# myproject/middlewares.py
import requests
from scrapy import Request
from scrapy.http import Response
from twisted.internet.threads import deferToThread

API_URL = "https://api.scarnox.com/api/tasks/create"


class PopularCaptchaMiddleware:
    def __init__(self, api_key: str, timeout: int):
        self.api_key = api_key
        self.timeout = timeout

    @classmethod
    def from_crawler(cls, crawler):
        return cls(
            api_key=crawler.settings.get("SCARNOX_API_KEY"),
            timeout=crawler.settings.getint("SCARNOX_TIMEOUT", 185),
        )

    def _solve(self, sitekey: str, page_url: str) -> str:
        r = requests.post(
            API_URL,
            headers={"Authorization": f"Bearer {self.api_key}"},
            json={
                "captcha_type": "PopularCaptchaTaskProxyless",
                "site_key": sitekey,
                "site_url": page_url,
            },
            timeout=self.timeout,  # runs in a worker thread, not the reactor
        )
        data = r.json()
        if data.get("errorId") != 0:
            raise RuntimeError(f"{data.get('errorCode')}: {data.get('errorDescription')}")
        return data["solution"]["token"]

    def process_response(self, request: Request, response: Response, spider):
        # Already solved, or a form submit we built ourselves: pass through.
        if request.meta.get("captcha_token") or request.meta.get("skip_captcha"):
            return response

        sitekey = response.css("[data-sitekey]::attr(data-sitekey)").get()
        if not sitekey:
            return response

        spider.logger.info("PopularCaptcha on %s, solving", response.url)

        def retry_with_token(token: str) -> Request:
            # Returning a Request reschedules it. dont_filter avoids the dupe filter.
            return request.replace(
                dont_filter=True,
                meta={**request.meta, "captcha_token": token},
            )

        return deferToThread(self._solve, sitekey, response.url).addCallback(retry_with_token)

Returning a Deferred from process_response is supported by Scrapy. The callback turns the token into a retried request, and Scrapy schedules it like any other.

The spider

# myproject/spiders/search.py
import scrapy
from scrapy.http import FormRequest


class SearchSpider(scrapy.Spider):
    name = "search"
    start_urls = ["https://search.example.com/search?q=web+scraping"]

    def parse(self, response):
        token = response.meta.get("captcha_token")
        if token and response.css("[data-sitekey]"):
            # Scrapy fills the form's own fields; we add the token.
            yield FormRequest.from_response(
                response,
                formdata={
                    "h-captcha-response": token,
                    "g-recaptcha-response": token,
                },
                callback=self.parse_results,
                meta={"skip_captcha": True},
                dont_filter=True,
            )
            return

        yield from self.parse_results(response)

    def parse_results(self, response):
        for item in response.css("div.result"):
            yield {
                "title": item.css("h3::text").get(),
                "url": item.css("a::attr(href)").get(),
            }

The selectors are placeholders. Replace them with the real ones from the site you are scraping.

The skip_captcha flag is what stops the middleware from solving the form submission again. Without it, every submit would cost another solve.

Run it

export SCARNOX_API_KEY=scarnox_YOURAPIKEY
scrapy crawl search -O results.json

Watch the log. A line such as PopularCaptcha on ..., solving means a challenge was found and a token was requested. If you see that line on every page, slow down. Frequent challenges are a sign the site has noticed the crawl.

Token timing

A PopularCaptcha token expires in about two minutes. The middleware requests the retry as soon as the token arrives, and the spider submits in the same chain of callbacks. Keep the chain short. Do not add slow downloads or long processing between the token and the form submit. For more on the window, see captcha token expiry.

Errors

A failed solve raises RuntimeError inside the thread. Scrapy will log it as a failed request. Failed solves are not charged, so retrying is cheap. For a retry policy with limits, use Scrapy's built-in RetryMiddleware and wrap _solve in a short loop. Our guide on captcha solver error handling lists the patterns that hold up in production.

Scrape responsibly

For the legal background, see is web scraping legal.

Pricing

Scarnox bills per successful solve, and failed solves are free. See current pricing for the rates. Start with free trial credit, no card needed.

Next steps