Scarnox · Blog

How to avoid captchas when scraping (without breaking the rules)

Most captchas during scraping are a reaction to traffic that looks automated. The cheapest way to deal with them is to stop triggering them. That means looking for a legitimate data source first, asking the site's rules what you may do, sending fewer and slower requests, and not fetching the same page twice. Solving a captcha is the fallback for the cases you cannot avoid, not the plan.

Check for an official API or feed first

Many sites that get scraped already offer a way to get the same data without a captcha: a public API, a data export, an RSS or sitemap feed, or a partner program. An official source is more reliable, less likely to change without notice and gives the site owner a say in how you use it. Look for:

If one of these covers your need, use it. You will avoid most captchas by not requesting HTML at all.

Respect robots.txt and the rate it asks for

robots.txt tells you which paths the site allows automated clients to fetch and, sometimes, how long to wait between requests. Python's standard library can read it for you:

import time
from urllib.robotparser import RobotFileParser
import requests

USER_AGENT = "ExampleBot/1.0 (+https://example.com/bot)"

parser = RobotFileParser()
parser.set_url("https://example.com/robots.txt")
parser.read()

delay = parser.crawl_delay(USER_AGENT) or 5  # fall back to a conservative 5 seconds
urls = [
    "https://example.com/products?page=1",
    "https://example.com/products?page=2",
]

last_request = 0.0
for url in urls:
    if not parser.can_fetch(USER_AGENT, url):
        print("skipped by robots.txt:", url)
        continue
    wait = delay - (time.monotonic() - last_request)
    if wait > 0:
        time.sleep(wait)
    resp = requests.get(url, headers={"User-Agent": USER_AGENT}, timeout=30)
    last_request = time.monotonic()
    print(url, resp.status_code)

If robots.txt disallows a path, the honest answer is to leave it alone or ask the owner. Routing around it with a browser does not change what the site asked for.

Slow down and spread requests

A steady, modest pace rarely triggers a challenge, while a burst almost always does. A few habits make a large difference:

Cache and do not refetch

Every request you do not make is a request the site does not have to handle. Store the pages you fetch with their date and a hash of the body, and only fetch again when the content has probably changed. Conditional requests with If-Modified-Since or ETag headers let the server answer "not modified" cheaply when it supports them.

Reuse sessions, send ordinary headers

A session that keeps cookies between requests behaves more like a visitor than a script that starts fresh every time. Use requests.Session() for each site, and send a clear, honest User-Agent that names your bot and a contact. Do not impersonate a browser to get past a rule. If a site needs JavaScript to show its content, use a browser tool for those pages, and keep the same rate limits.

When a captcha still appears

Some sites will show a captcha no matter how carefully you behave. Detect it in code and stop that job rather than hammering the page. The detection and the optional solve for PopularCaptcha are covered in web scraping captchas.

Plan the crawl before you start

A crawl that fetches everything every day is the easiest way to trigger a captcha. Plan the work first:

A plan like this lowers the number of requests, which is the biggest lever you have.

Write the plan down as a short config file or a table: the sites, the sections, the refresh interval and the daily request budget for each. When the budget is spent, the job stops until the next window. Treating the budget as a hard limit, rather than a guideline, is what keeps a crawler from drifting into the traffic pattern that triggers captchas in the first place. Review the budget every few weeks, because a site that grows may also grow its traffic, and what was polite last quarter may not be polite now.

Scrape responsibly

Next steps