AI bots
CCBot: what it is, its IP ranges, and how to block it
CCBot is Common Crawl's crawler for AI training.
| Vendor | Common Crawl |
|---|---|
| What it is for | AI training |
| In its User-Agent | CCBot |
| Its vendor's address list | https://index.commoncrawl.org/ccbot.json |
| IP ranges | 5 |
It takes pages to train AI models and sends nobody back to your site. Bot Protection blocks it by default; you can let it through.
How to verify it
Any bot can write CCBot in its User-Agent. A request is CCBot only when its address is in Common Crawl's published list below: check the address, not the name.
How to block it
With robots.txt, you ask it to stay out:
User-agent: CCBot Disallow: /
robots.txt is a request, not a block, and it cannot tell the real crawler from a bot wearing its name. Bot Protection verifies CCBot by its address on every request and answers it as you choose on the Set-up page, per kind or for CCBot alone; a bot that only wears the name is judged like any other bot.
Its IP ranges
From Common Crawl's own list, downloaded again every day.
IPv4
3.41.188.32/29 18.97.9.168/29 18.97.14.80/29 18.97.14.88/30
IPv6
2600:1f28:365:8000::/56