Bot directory
The crawlers and AI agents Bot Protection verifies
Each bot below says who it is, and its vendor publishes how to prove it: the addresses it crawls from, or the keys it signs its requests with. Anyone can write a bot's name; only the address or the signature proves it. These are the vendors' own lists, read again every day, the same ones Bot Protection verifies every request against.
Search engines
| Bot | Vendor | Verified by | By default |
|---|---|---|---|
Googlebot | Published addresses | Let through | |
Applebot | Apple | Published addresses | Let through |
DuckDuckBot | DuckDuckGo | Published addresses | Let through |
AI assistants and AI search
| Bot | Vendor | Verified by | By default |
|---|---|---|---|
OAI-SearchBot | OpenAI | Published addresses | Let through |
ChatGPT-User | OpenAI | Published addresses | Let through |
OAI-AdsBot | OpenAI | Published addresses | Let through |
Claude-User | Anthropic | Published addresses | Let through |
Claude-SearchBot | Anthropic | Published addresses | Let through |
PerplexityBot | Perplexity | Published addresses | Let through |
Perplexity-User | Perplexity | Published addresses | Let through |
Google-CloudVertexBot | Published addresses | Let through | |
Google-Agent | Published addresses | Let through | |
DuckAssistBot | DuckDuckGo | Published addresses | Let through |
MistralAI-User | Mistral AI | Published addresses | Let through |
MistralAI-Index | Mistral AI | Published addresses | Let through |
AI agents
| Bot | Vendor | Verified by | By default |
|---|---|---|---|
ChatGPT agent | OpenAI | Web Bot Auth signature | Let through |
Scraping and AI training crawlers
| Bot | Vendor | Verified by | By default |
|---|---|---|---|
GPTBot | OpenAI | Published addresses | Block |
ClaudeBot | Anthropic | Published addresses | Block |
CCBot | Common Crawl | Published addresses | Block |
SEO tools
| Bot | Vendor | Verified by | By default |
|---|---|---|---|
AhrefsBot | Ahrefs | Published addresses | Let through |
AhrefsSiteAudit | Ahrefs | Published addresses | Let through |
Link previews
| Bot | Vendor | Verified by | By default |
|---|---|---|---|
TelegramBot | Telegram | Published addresses | Let through |
Uptime monitors
| Bot | Vendor | Verified by | By default |
|---|---|---|---|
UptimeRobot | UptimeRobot | Published addresses | Let through |
Pingdom.com_bot | Pingdom | Published addresses | Let through |
Better Uptime Bot | Better Stack | Published addresses | Let through |
Internet scanners
| Bot | Vendor | Verified by | By default |
|---|---|---|---|
CensysInspect | Censys | Published addresses | Block |
Expanse | Palo Alto Networks | Published addresses | Block |
By default is what Bot Protection answers a verified bot until you choose: on the Set-up page you let through or block each kind, or each bot on its own.
Named, never verified
These crawlers name themselves in their User-Agent, but their vendors publish no addresses or keys to prove it, so the name alone proves nothing: a request that wears one is judged like any other visitor.
| Bot | Vendor | What it is for |
|---|---|---|
Amazonbot | Amazon | AI training |
Amzn-SearchBot | Amazon | AI search and answers |
Amzn-User | Amazon | AI search and answers |
meta-externalagent | Meta | AI training |
meta-externalfetcher | Meta | AI search and answers |
Meta-WebIndexer | Meta | AI search and answers |
bingbot | Microsoft | Search engine |
Bytespider | ByteDance | AI training |
cohere-ai | Cohere | AI training |
AI2Bot | Allen Institute for AI | AI training |
SemrushBot | Semrush | SEO tool |
MJ12bot | Majestic | SEO tool |
facebookexternalhit | Meta | Link preview |
Twitterbot | X | Link preview |
Slackbot | Slack | Link preview |
LinkedInBot | Link preview | |
Discordbot | Discord | Link preview |