{"$schema":"https://utilix.app/crawlers.schema.json","version":1,"source":"https://utilix.app/crawlers.json","license":"CC0-1.0","description":"Web crawler user-agent tokens, grouped by what each one is actually for. Blocking a training crawler and blocking a search crawler are different decisions with opposite consequences, and most published lists predate the operators splitting them. Every entry is read from the operator's own documentation, except where documented is false.","counts":{"crawlers":24,"operators":9,"byPurpose":{"search":6,"training":6,"operational":5,"user-triggered":5,"dataset":2}},"sources":{"Google":"https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers","OpenAI":"https://developers.openai.com/docs/bots","Anthropic":"https://support.claude.com/en/articles/8896518","Perplexity":"https://docs.perplexity.ai/guides/bots","Apple":"https://support.apple.com/en-us/119829","Amazon":"https://developer.amazon.com/amazonbot","Meta":"https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/","Common Crawl":"https://commoncrawl.org/ccbot","ByteDance":""},"crawlers":[{"token":"Googlebot","operator":"Google","purpose":"search","purposeLabel":"Search visibility","summary":"Google Search, Images, Video, News and Discover.","ifBlocked":"You leave Google Search. This is almost never what someone wants, and it is the single most expensive line anyone puts in this file by accident.","caveat":null,"documented":true},{"token":"Google-Extended","operator":"Google","purpose":"training","purposeLabel":"Model training","summary":"Gemini model training and grounding.","ifBlocked":"Your content stops being used to train Gemini and stops being pulled in as grounding at prompt time. Search ranking is untouched.","caveat":"Not a crawler. It fetches nothing of its own — it is a control token attached to crawling Google already does, so you will never see it in your access logs.","documented":true},{"token":"Google-CloudVertexBot","operator":"Google","purpose":"operational","purposeLabel":"Operational","summary":"Crawls sites on their own owner's request for Vertex AI agents.","ifBlocked":"No effect on Google Search. Only relevant if someone is building a Vertex AI agent over your site.","caveat":null,"documented":true},{"token":"GoogleOther","operator":"Google","purpose":"operational","purposeLabel":"Operational","summary":"Generic internal fetching, research and development.","ifBlocked":"Nothing user-facing. Tied to no specific product.","caveat":null,"documented":true},{"token":"GPTBot","operator":"OpenAI","purpose":"training","purposeLabel":"Model training","summary":"Crawls content that may train OpenAI foundation models.","ifBlocked":"Signals that your content should not be used for training. Your visibility in ChatGPT search is unaffected — that is a different token.","caveat":null,"documented":true},{"token":"OAI-SearchBot","operator":"OpenAI","purpose":"search","purposeLabel":"Search visibility","summary":"Surfaces sites in ChatGPT's search results.","ifBlocked":"You stop appearing in ChatGPT search answers, though you can still show up as a plain navigational link. Allow this one if you want ChatGPT traffic.","caveat":"Changes take roughly 24 hours to propagate after you edit robots.txt.","documented":true},{"token":"ChatGPT-User","operator":"OpenAI","purpose":"user-triggered","purposeLabel":"User-triggered fetches","summary":"Visits a page because a ChatGPT user asked about it.","ifBlocked":"Little, reliably. OpenAI states robots.txt rules may not apply here, since a person initiated the request rather than a crawl.","caveat":"Not used for automatic crawling and not what decides whether you appear in ChatGPT search. Use OAI-SearchBot for that.","documented":true},{"token":"OAI-AdsBot","operator":"OpenAI","purpose":"operational","purposeLabel":"Operational","summary":"Safety-checks landing pages submitted as ChatGPT ads.","ifBlocked":"Only matters if you advertise on ChatGPT. It visits submitted ad landing pages only, and does not feed model training.","caveat":null,"documented":true},{"token":"ClaudeBot","operator":"Anthropic","purpose":"training","purposeLabel":"Model training","summary":"Collects web content that may contribute to model training.","ifBlocked":"Signals that your future material should be excluded from Anthropic training datasets.","caveat":null,"documented":true},{"token":"Claude-SearchBot","operator":"Anthropic","purpose":"search","purposeLabel":"Search visibility","summary":"Indexes content to improve Claude's search results.","ifBlocked":"Your content stops being indexed for search, which reduces your visibility in Claude's search answers.","caveat":null,"documented":true},{"token":"Claude-User","operator":"Anthropic","purpose":"user-triggered","purposeLabel":"User-triggered fetches","summary":"Fetches a page when a Claude user asks about it.","ifBlocked":"Claude stops retrieving your content in response to user questions, which can reduce visibility in user-directed search.","caveat":null,"documented":true},{"token":"PerplexityBot","operator":"Perplexity","purpose":"search","purposeLabel":"Search visibility","summary":"Surfaces and links sites in Perplexity results.","ifBlocked":"You stop appearing in Perplexity answers. Note this bot is explicitly not used for foundation model training, so blocking it buys you no training protection.","caveat":"Changes take up to 24 hours to take effect.","documented":true},{"token":"Perplexity-User","operator":"Perplexity","purpose":"user-triggered","purposeLabel":"User-triggered fetches","summary":"Visits a page to answer a specific user question.","ifBlocked":"Generally nothing. Perplexity documents that this fetcher ignores robots.txt, because a user requested the fetch.","caveat":"Documented as ignoring robots.txt. A rule here is a statement of preference, not a control.","documented":true},{"token":"Applebot","operator":"Apple","purpose":"search","purposeLabel":"Search visibility","summary":"Powers Spotlight, Siri and Safari search.","ifBlocked":"You disappear from search across Apple's ecosystem. Rarely worth it.","caveat":null,"documented":true},{"token":"Applebot-Extended","operator":"Apple","purpose":"training","purposeLabel":"Model training","summary":"Opt out of Apple foundation model training.","ifBlocked":"Your content stops training Apple's generative models. Applebot keeps crawling and you stay discoverable in Spotlight, Siri and Safari.","caveat":"Blocking this does not remove you from Apple's AI answers. To limit that, add a nosnippet meta tag to the content itself.","documented":true},{"token":"Amazonbot","operator":"Amazon","purpose":"training","purposeLabel":"Model training","summary":"Improves Amazon products and may train Amazon AI models.","ifBlocked":"Your content stops feeding Amazon's model training. Alexa search visibility is a separate token and is unaffected.","caveat":null,"documented":true},{"token":"Amzn-SearchBot","operator":"Amazon","purpose":"search","purposeLabel":"Search visibility","summary":"Makes content eligible to appear in Alexa and Amazon search.","ifBlocked":"You stop being eligible for Amazon search experiences such as Alexa. Amazon states this bot does not crawl for generative training, so blocking it buys no training protection.","caveat":"If your file never names it but allows other search bots, it follows those rules instead. Silence is not a block here.","documented":true},{"token":"Amzn-User","operator":"Amazon","purpose":"user-triggered","purposeLabel":"User-triggered fetches","summary":"Fetches live pages to answer an Alexa question.","ifBlocked":"Possibly nothing. Amazon documents that because a user initiates the fetch, it may not follow all robots.txt directives.","caveat":"Also stated not to crawl for model training, so a rule here is about live retrieval only.","documented":true},{"token":"Meta-ExternalAgent","operator":"Meta","purpose":"training","purposeLabel":"Model training","summary":"Crawls for foundation model training and direct indexing.","ifBlocked":"Your content stops being collected for Meta's AI training.","caveat":"Meta describes this one as doing both training and indexing, so unlike Google or OpenAI there is no separate search token to keep open alongside it.","documented":true},{"token":"Meta-ExternalFetcher","operator":"Meta","purpose":"user-triggered","purposeLabel":"User-triggered fetches","summary":"Fetches a link on a user's request, including for agentic tasks.","ifBlocked":"Likely nothing. Meta states this crawler may bypass robots.txt rules, because a person asked for the page.","caveat":"Documented as able to ignore robots.txt. Treat a rule here as a preference.","documented":true},{"token":"FacebookExternalHit","operator":"Meta","purpose":"operational","purposeLabel":"Operational","summary":"Builds the link preview when someone shares your page.","ifBlocked":"Shared links lose their title, description and thumbnail across Facebook, Instagram and Messenger. Rarely worth it.","caveat":"May bypass robots.txt during security or integrity checks, such as scanning for malware.","documented":true},{"token":"meta-externalads","operator":"Meta","purpose":"operational","purposeLabel":"Operational","summary":"Crawls for Meta's advertising products.","ifBlocked":"Only relevant if your pages are advertised on Meta platforms.","caveat":null,"documented":true},{"token":"CCBot","operator":"Common Crawl","purpose":"dataset","purposeLabel":"Open corpora","summary":"Builds the open web corpus that many models are trained on.","ifBlocked":"Your content stops entering future Common Crawl snapshots. That is the widest single opt-out available here, since the corpus feeds many organisations at once — but it is not retroactive, and archives already published keep what they have.","caveat":"Common Crawl warns that other crawlers impersonate CCBot, so verify by reverse DNS against crawl.commoncrawl.org before trusting a log line.","documented":true},{"token":"Bytespider","operator":"ByteDance","purpose":"dataset","purposeLabel":"Open corpora","summary":"Collects content for ByteDance search and recommendations.","ifBlocked":"Unclear. ByteDance publishes no crawler documentation, and independent log analyses report Bytespider fetching robots.txt and then crawling disallowed paths anyway.","caveat":"The only entry here not taken from an operator's own page, because there is no such page. If its volume is actually hurting you, block it at the server or CDN rather than trusting this file.","documented":false}]}