Scarnox · Blog

Price monitoring with web scraping: a practical guide

Price monitoring means checking the price of a product on a set of sites, on a schedule, and recording what you see so you can compare over time. Retailers, marketplaces and analysts all use it. Price scraping is also one of the most common ways a scraper meets a captcha, so it is worth getting the design right. This guide covers what to monitor, how to parse numbers from pages that were never meant to be parsed, how to store and alert on changes, and how to stay within a site's rules while you do it.

Start with a source you are allowed to use

Before writing any code, check whether the site offers an official way to get the data: a product feed, an API, or a partner programme. These are more reliable than scraping, and they remove most of the legal and technical risk. Scraping is the fallback for public pages where no official route exists, and the terms and robots.txt still apply. The legal overview covers the general questions.

Architecture in five parts

A reliable monitor has five parts, and keeping them separate makes it easier to debug:

  1. Scheduler. Decides which product pages to visit and when. Spread visits out over the day rather than checking everything at the same minute.
  2. Fetcher. Downloads the page, with polite pacing, retries and a cache.
  3. Parser. Pulls out the price, the currency context and the stock status.
  4. Store. Keeps every observation with a timestamp, so you have a history.
  5. Alerter. Compares the new value with the last one and notifies you when it crosses a threshold you set.

Parse prices from messy pages

Prices appear in many forms: with thousands separators, with decimal commas, with a label before the number, or split across several elements. Keep the parser small and test it against real examples. Here is a version that assumes the common format with commas for thousands and a dot for decimals. Normalise other formats before you call it.

import re
from decimal import Decimal, InvalidOperation

PRICE_PATTERN = re.compile(r"\d{1,3}(?:,\d{3})*(?:\.\d{1,2})?|\d+(?:\.\d{1,2})?")

def parse_price(text: str) -> Decimal | None:
    """Return the first number in the text as a Decimal, or None."""
    match = PRICE_PATTERN.search(text.replace(" ", " "))
    if not match:
        return None
    try:
        return Decimal(match.group(0).replace(",", ""))
    except InvalidOperation:
        return None

print(parse_price("Now 1,249.50 in stock"))  # Decimal('1249.50')
print(parse_price("Sold out"))               # None

Use Decimal, not float, for anything you will compare or total. Floating-point rounding can make two equal prices look different. Keep the raw text next to the parsed value, so you can fix a parser bug later without re-fetching every page.

Store history and detect changes

A single price is a snapshot. The value of a monitor is the history. Store one row per observation, with the product ID, the source, the timestamp, the parsed value and the raw text. Then alerting is a simple comparison against the previous row. Alert on meaningful changes only, such as a drop beyond a set percentage, or a product going out of stock, so you do not get flooded with noise.

Handle variants and bundles

A product page often shows several prices: one per size or colour, a bundle price, a member price, or a price that appears only after a selection is made. Decide before you build what the monitor should record. A common approach is to store the variant identifier alongside the price, so that a change in one size does not look like a change in the whole product. Be careful with "from" prices and crossed-out original prices, because a parser that picks up the wrong number will raise false alerts. Store the label text next to each value, so you can check what was read when an alert looks wrong.

Schedule politely

Check each product at an interval that matches how often its price actually moves. A product that changes daily does not need an hourly check. Add jitter to each visit, cap the number of requests per site per hour, and cache results for the interval you chose. These habits keep your load low and your data consistent.

When the page blocks you

Price pages are frequent targets for automated access, so you may see a block, a rate limit or a captcha. Back off when you see a 429 or 503, and stop visiting a site that keeps refusing you. If a captcha appears on a page you are allowed to automate, the live captcha type at Scarnox is PopularCaptcha. The PopularCaptcha Python guide shows the token flow. Read scrape without getting blocked before you add more machinery, because pacing usually solves more than any extra tool.

Some pages render their prices with JavaScript. For those, follow the approach in scraping JavaScript websites.

Scrape responsibly

Next steps

Start with free trial credit, no card needed: create an account and try the PopularCaptcha flow on a page you are allowed to automate. For current rates, see the pricing page.