Scarnox · Blog

How to scrape JavaScript-rendered websites

Many modern sites send an almost empty HTML document and build the content in the browser with JavaScript. If your scraper fetches the page and finds no products, no prices and no article text, the content is there, but it arrives later. This guide shows how to tell which kind of page you have, how to find a faster route than a browser where one exists, and how to scrape rendered pages with Playwright in Python. It ends with a note on what to do when a rendered page shows a captcha.

Step 1: check whether the content is in the HTML

Open the page with view-source: in your browser, or fetch it with a plain HTTP request, and search the source for a piece of text you can see on screen. Three outcomes are common:

The text is there. The page is server-rendered. A plain HTTP client and an HTML parser are enough, and you should use them, because they are faster and simpler.
The text is missing, but a JSON blob is present. Many sites embed their state in a script tag, such as __NEXT_DATA__. Parse that JSON directly.
The text is missing and there is no blob. The page loads its content with further requests. Look at the Network tab in DevTools, filtered to Fetch/XHR, while the page loads.

Step 2: call the JSON endpoint if one exists

When a page fetches its data from an endpoint, calling that endpoint directly is usually the best option. It is smaller, faster and more stable than the rendered page. Copy the request URL, check which headers the site requires, and try it from a script. Do this only where the site's terms allow it, and respect any rate limits or authentication the site expects. An undocumented endpoint can change without notice, so write your parser to fail loudly when the shape of the response changes.

Step 3: render the page when you must

If the content truly needs the browser, drive a headless browser. Playwright is a good choice for Python. The example below opens a product listing, waits for the content to appear, and reads the text from each card:

from playwright.sync_api import sync_playwright

URL = "https://example.com/products"  # a page you are allowed to scrape

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto(URL, wait_until="domcontentloaded", timeout=60_000)
    # Wait for the element that proves the data has arrived.
    page.wait_for_selector(".product-card", timeout=30_000)
    titles = page.locator(".product-card h2").all_inner_texts()
    browser.close()

for title in titles:
    print(title)

Waiting for a specific selector is far more reliable than a fixed sleep. A fixed pause is either too short on a slow day or wastes time on a fast one. Pick the element that only appears once the data has loaded.

Step 4: handle infinite scroll and pagination

Some pages load more items as you scroll. Scroll to the bottom, wait for the item count to grow, and stop when it no longer grows. Pages with numbered pagination are simpler: find the next-page link, follow it, and stop at the last page. Always set a maximum number of pages or scrolls, so a bug cannot keep your crawler running forever.

def scroll_until_done(page, selector: str, max_rounds: int = 20) -> int:
    previous = -1
    for _ in range(max_rounds):
        count = page.locator(selector).count()
        if count == previous:
            break
        previous = count
        page.mouse.wheel(0, 4000)
        page.wait_for_timeout(1500)  # give the next batch time to load
    return previous

Step 5: keep the browser economical

Rendering is expensive. Block images, fonts and media that you do not need, and reuse one browser for many pages rather than launching a new one each time. Close pages and contexts when you are done, so memory does not climb across a long run.

def block_heavy_assets(route):
    if route.request.resource_type in ("image", "font", "media"):
        route.abort()
    else:
        route.continue_()

page.route("**/*", block_heavy_assets)

Step 6: when a rendered page shows a captcha

Sometimes the rendered page presents a captcha before it shows any content. If that captcha is PopularCaptcha, the live captcha type at Scarnox, you read the sitekey from the page, request a token, and place the token in the form field that the site expects. The PopularCaptcha Python guide walks through that step. Other widgets are not live yet, so check captcha types explained before you plan around one.

Be careful about the browser itself. A headless browser that looks clearly automated can trigger more checks, not fewer. Read headless browser detection for the signals to watch, and prefer slower, fewer page loads.

Scrape responsibly

Check robots.txt and the terms of the site before you render or call its endpoints.
Keep rate limits low, and cache results so you fetch each page once.
Do not scrape personal data without a lawful basis.
If a site restricts access, stop. Do not try to get past the restriction.

Next steps

Start with free trial credit, no card needed: create an account and test the PopularCaptcha flow on a page you own. For current rates, see the pricing page.