Web scraping is the copying of a site's content or data by automation. OWASP defines it as collecting accessible data for later use, "such as exploitation of proprietary data or price-fixing", and lists the names it goes by: content scraping, data aggregation, harvesting, screen scraping, and now LLM and AI scrapers. Some scraping is welcome, since a search engine is a scraper that sends you readers. The question is never "is it a bot" but "did you invite it".
What scraping costs you
- Your data, used against you. A competitor reads your prices to undercut them, a copy site republishes your articles, an aggregator resells your listings.
- The bill for serving it. The Wikimedia Foundation reported in April 2025 that bots made about 35% of its pageviews but at least 65% of its most expensive traffic, the requests that miss its caches and reach its core datacenter.
- Accounts made for it. OWASP notes that some scraping runs through fake or stolen accounts, to reach what only members see. That makes your sign-up part of the problem: see how to stop fake sign-ups.
robots.txt is a request, not a lock
Say in robots.txt what crawlers may read; the large, honest ones follow it. But the standard that defines it, RFC 9309, says it plainly: "these rules are not a form of access authorization". A scraper that wants your prices does not read it, so it is where to start, not where to stop.
Let the crawlers you want through, by proof
Anyone can write Googlebot in a User-Agent. Google tells site owners to check the claim instead: a reverse and forward DNS lookup on the address, or a match against the address ranges Google publishes. Bot Protection does that check for you: when a crawler that names itself comes from its vendor's published addresses, the answer names it in crawler, with its kind. Search engines, AI assistants and link previews go through unless you block them; AI training crawlers are blocked unless you let them through, and you can choose for each crawler on its own. Every crawler it verifies is in the bot directory. One that only borrows the name is judged like any other visitor.
Ask where the data is
Scrapers go where the value is, so put the check there, not on every page: the product and price pages, search results, listings, the articles you sell, and the JSON your own pages call. Your server asks Bot Protection before it answers, and acts on the class:
bot_high: refuse, or serve less, such as an article's teaser or a page without the prices.bot: slow it, serve it from the cache, or send it through the challenge page.unknown,human,human_high, and the crawlers you invited: the full page.
The home page and the pages that only bring people in can stay open to everyone. Asking only where the data is keeps the check off the paths that do not need it, and off your bill. Publishers have their own case in bot protection for news sites, and AI crawlers and agents in AI agents and LLM scrapers.
See it before you block it
Start in monitor mode. Nobody is stopped, and each answer says what block mode would have done, so you learn how much of your price pages or your archive is read by scripts before you change what anyone sees.
Keep it fail-open
A page that waits on a bot check is slow when the check is slow. Use a short timeout and serve the page on any error: a scraper that gets a page during an outage costs less than readers who get none. More in why a bot check must fail open.
Sources
Checked on 28 October 2026.