Scarnox · Blog

Is web scraping legal? A general overview

Is web scraping legal? The honest answer is that it depends. It depends on where you and the site are located, on what kind of data you collect, on how you collect it, on what the site's terms say, and on what you do with the result. This page gives general information about the main factors that courts and regulators tend to look at. It is not legal advice. For a decision that matters to your project or your business, talk to a qualified lawyer in the relevant jurisdiction.

The short version

Scraping publicly visible pages is not automatically illegal. Several things raise the risk: collecting personal data, ignoring a clear prohibition, copying large amounts of protected content, getting past access controls, and causing harm to the site. The more of these apply, the more careful you need to be.

Factor 1: what data you collect

Facts and public information are treated differently from creative works and from personal data.

Publicly available facts, such as a published price or a stock level, are generally treated more permissively than copyrighted text or images.
Copyrighted content, such as articles, photographs and databases, can be protected even when it is public. Copying it wholesale for reuse carries more risk than extracting specific facts.
Personal data, meaning information that identifies a person, is subject to data protection laws in many places. The GDPR in the European Union and similar laws elsewhere place obligations on how it is collected, stored and used, even when it was public. Ask whether you need the personal data at all.

Factor 2: the site's terms and robots.txt

Many sites publish terms of service that forbid automated access. Whether a breach of those terms has legal consequences depends on the jurisdiction and the facts, and courts have reached different conclusions in different cases.

The robots.txt file is a machine-readable signal of what the owner wants crawlers to do. It is not a law in itself, but ignoring it can be used as evidence that you acted unreasonably, and a well-run project respects it by default.

Factor 3: how you collect

The method matters as much as the target. Sending a modest number of requests, spaced out, is very different from flooding a server and degrading service for other users. Some jurisdictions treat unauthorised access to computer systems as a criminal offence. Getting past a login, a paywall or an access control that the owner has put in place raises the risk considerably, and it can cross a legal line that ordinary scraping does not.

For that reason, Scarnox is built for legitimate automation: testing your own sites, accessibility work, and collecting public data within a site's rules. It is not intended for use against access controls, and the captcha steps in these guides apply to flows you are allowed to automate.

Factor 4: what you do with the data

Collecting data is one step. Republishing it, selling it, or using it to compete unfairly can raise separate issues, including copyright, database rights, unfair-competition rules and contract claims. Keep a clear record of where the data came from, why you collected it, and how long you keep it.

Why the answer changes from place to place

Law on this topic is not uniform. Some countries have specific rules on database rights, others rely on general contract and computer-misuse law, and data protection rules differ widely. Court decisions on scraping have also differed from one case to the next, and the outcome often turned on details such as the kind of data, whether a login was involved, and whether the scraper ignored a clear demand to stop. Because of that, a rule that applies in one place may not apply in another. Treat any general summary, including this one, as a starting point for a conversation with a professional who knows your situation.

A practical checklist

Before you start a scraping project, you can work through these questions with your team or counsel:

Is the data public, and is it factual rather than creative or personal?
Have you read the site's terms and robots.txt, and does your use fit them?
Do you need personal data at all? If so, do you have a lawful basis for it?
Are you staying within the access the site offers, without bypassing controls?
Is your request rate low enough that normal users are not affected?
Do you have a plan to delete data you no longer need?
Would you be comfortable explaining your method to the site owner?

If you answer "no" to several of these, stop and get advice before you continue.

Scrape responsibly

Read robots.txt and the terms before you scrape a site.
Keep your request rate modest, and cache what you collect.
Limit the personal data you store, and protect what you keep.
Do not use Scarnox, or any other tool, to get past a restriction the owner has put in place.

Next steps

Start with free trial credit, no card needed: create an account to explore the PopularCaptcha flow on a site you are allowed to automate. For current rates, see the pricing page.