Scarnox · Blog

AWS WAF captcha explained and how to handle it

AWS WAF captcha is a challenge served by Amazon's Web Application Firewall when a site owner has configured it to check visitors. A request that the firewall considers suspicious receives a challenge page instead of the content. The visitor then has to pass a check before the real page loads. If you run into it while testing or collecting data, you need to know how to recognise it, what the site owner controls, and what Scarnox can do today. This guide covers all three.

What the AWS WAF captcha is

AWS WAF is a rules engine that sits in front of a web application, on Amazon CloudFront, an Application Load Balancer or an API Gateway. Rules can allow, block, count or challenge a request. A challenge can be a silent JavaScript check or a visible captcha puzzle, depending on how the rule is set up.

The important point is that the decision is made by the site owner. The firewall does not know your scraper's purpose. It only scores the traffic it sees: the rate of requests, the IP reputation, the headers and the behaviour of the client.

How to recognise it in code

AWS WAF challenges are often identifiable by their responses. A common signal is an x-amzn-waf-action response header, and a challenge is often served with HTTP status 202 before the real page appears. Treat these as signals, not proof, because sites configure the firewall differently:

import requests

def looks_like_aws_waf(resp: requests.Response) -> bool:
    # A header signal, not proof. Confirm against the page you are testing.
    return "x-amzn-waf-action" in resp.headers

resp = requests.get("https://example.com/products", timeout=30)
if looks_like_aws_waf(resp):
    print("Challenge served. Stop, slow down, and check your request rate.")
else:
    print("Status", resp.status_code)

Also look for a script loaded from an AWS WAF challenge path in the page source, and for a page that shows no content until a script runs. A plain requests call will often receive the challenge page rather than the data. This is expected.

What the site owner can do

If you run the site, the AWS WAF console lets you tune the rules that trigger a challenge. The most useful options are:

If you are the one testing your own site, the cleanest fix is an allowlist for your test runners. Do not try to defeat the challenge on a site you do not control. That is the owner's protection, and it is the first thing that will break your access.

Where Scarnox support stands

So if a site you are allowed to work with shows an AWS WAF challenge today, the options are the ones above: slow down, use an allowlist if the site is yours, or ask the site owner for an API or data feed. You can create an account and start with free trial credit, no card needed, to set up your PopularCaptcha flow now and be ready when AWS WAF support goes live. Current pricing is on the pricing page.

Test your own site behind AWS WAF

If you are the site owner and want automated tests to pass the firewall without weakening it for real visitors, use the firewall's own tools rather than working around the challenge:

These steps keep the protection in place and make the test traffic predictable, which is the outcome you actually need. Review the rules after each change to your test setup, because a rule that was correct for last month's runners may now block them.

Scrape responsibly

If your reason for hitting an AWS WAF page is collecting data:

Next steps