# AI Crawler List: Every Bot That Reads Your Site (2026)

> Ten AI crawlers fetch your pages in 2026. This is the live table of their operators, robots.txt tokens, and how to verify each one. Plus when blocking is the right call.

Source: https://geoplaybooks.com/ai-crawlers/

# AI Crawler List: Every Bot That Reads Your Site

At least ten AI crawlers fetch public pages in 2026. The main ones are GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic, and PerplexityBot from Perplexity. CCBot and cohere-ai are here too. Each carries its own robots.txt token. The user-agent string is easy to fake, so verify a real request by its IP.

This AI crawler list does two jobs. It gives you the whole set in one table. It routes you to the per-bot card when you need exact directives. For the deepest single card, see the [ClaudeBot](/ai-crawlers/claudebot/) reference.

## Which AI crawlers visit your site

Google's AI Overview for "block ai crawlers," captured July 9, 2026, ships one sample robots.txt block. It names six tokens:

- `GPTBot`

- `anthropic-ai`

- `Claude-Web`

- `cohere-ai`

- `CCBot`

- `Google-Extended`

That is the floor, not the full roster. No marketer registry lists them all. The set is scattered across operator docs. This page assembles it.

One operator often runs several crawlers. OpenAI's bots page documents `GPTBot` for bulk crawling and `OAI-SearchBot` for answer-time search. Each setting is independent. Anthropic runs `ClaudeBot` plus two legacy tokens. The consequence is blunt. Block `GPTBot`, and `ClaudeBot` still reads you.

## The live user-agent table

This AI crawler list runs to ten crawlers and ten tokens. One honest caveat: the user-agent string proves nothing. Anyone can send it. So match on the token. Then confirm the request by IP. IP verification is the only proof that holds.

Check the source IP against the operator's published ranges. Or let Cloudflare bot management flag the verified bots. Strings and IP ranges rotate. Verify an exact string against the operator's live docs.

One row needs a footnote. `Google-Extended` is not a crawler. It is a control token. It decides whether pages Googlebot already fetched may feed Gemini and Vertex AI. Blocking it stops the AI use, not the fetch. For the full ClaudeBot card, read the [ClaudeBot](/ai-crawlers/claudebot/) reference.

## Block or allow? The orientation

A review of a few thousand US and UK sites found 27% blocking at least one major LLM crawler. Most of it was an accident.

The block lived at the CDN, WAF, or edge, not in robots.txt (r/aeo, February 18, 2026). So the first question is not whether to block. It is whether you already are.

*Block-or-allow decision path. A review of thousands of sites found 27% already block a major LLM crawler, mostly by accident (r/aeo, Feb 2026).*

> BLOCK OR ALLOW The block-or-allow call comes down to one question 27% already block a major LLM crawler — mostly by accident THE ORIENTATION Are AI answers a channel for you? Yes Allow the crawlers. Then audit you're not blocking by accident — the 27% trap. No Block on purpose, or charge. Content is the product; pay-per-crawl keeps the door open at a price. Either way: verify every request by IP, not by user-agent string. Block-or-allow decision path. Source: r/aeo, Feb 18, 2026.

Blocking is a legitimate choice, and this page will not talk you out of it. The exchange is one-sided. AI crawlers take content and generate answers from it. They send little or no referral traffic back. A WordPress origin still pays PHP, plugin, and database load on every request (r/Wordpress, April 28, 2026).

That is a fair case for publisher control. The load can spike hard. One owner logged Meta's crawler making 7.9 million requests and burning 900-plus GB in 30 days (r/webdev, April 6, 2026). That spike is why owners start looking. And the directives are only a request. The AI Overview notes that non-compliant bots "may ignore" robots.txt.

> 7.9 million requests from one Meta crawler 900-plus GB bandwidth burned by that crawler 30 days logging window

Two facts sharpen the call. Training-time crawling and answer-time retrieval are different levers. "Sites that block AI training crawlers mostly ignore the answer time bots" (Hacker News, July 7, 2026). So check which lever your rule pulls.

There is also a middle path: pay-per-crawl. Cloudflare offers a model for it. One Caddy plugin charges crawlers real USDC per request (Hacker News, February 27, 2026). It keeps the door open at a price.

The verdict, no hedge. Are AI answers a channel for you? Allow the crawlers. Then audit that you are not blocking them by accident. That is the 27% trap.

Is your content the product, and the trade not worth it? Block on purpose, or charge for access. Either way, verify each request by IP. Do not let a stale robots.txt decide for you. When you allow, use [llms.txt](/llms-txt/) as the explicit invitation.

## Every crawler we track

Every crawler on this AI crawler list gets its own reference card. You get the exact token, the IP check, the robots.txt directives, and the observed crawl rate. The [ClaudeBot](/ai-crawlers/claudebot/) card is live now. GPTBot, OAI-SearchBot, PerplexityBot, and a how-to-block guide are next.

## Every crawler we track

- [ClaudeBot: What It Is, How to Verify It, What Blocking It Costs](/ai-crawlers/claudebot/) — ClaudeBot is Anthropic's web crawler.

> {"@context": "https://schema.org", "@type": "Article", "headline": "AI Crawler List: Every Bot That Reads Your Site", "description": "A live reference of the AI crawlers that fetch public pages in 2026, their operators, and robots.txt tokens.", "url": "/ai-crawlers/", "datePublished": "2026-07-10", "dateModified": "2026-07-10", "author": {"@type": "Person", "name": "Editorial Team"}}

> {"@context": "https://schema.org", "@type": "ItemList", "name": "AI crawlers tracked", "numberOfItems": 10, "itemListElement": [{"@type": "ListItem", "position": 1, "name": "GPTBot"}, {"@type": "ListItem", "position": 2, "name": "OAI-SearchBot"}, {"@type": "ListItem", "position": 3, "name": "ChatGPT-User"}, {"@type": "ListItem", "position": 4, "name": "ClaudeBot", "url": "/ai-crawlers/claudebot/"}, {"@type": "ListItem", "position": 5, "name": "anthropic-ai"}, {"@type": "ListItem", "position": 6, "name": "Claude-Web"}, {"@type": "ListItem", "position": 7, "name": "PerplexityBot"}, {"@type": "ListItem", "position": 8, "name": "Google-Extended"}, {"@type": "ListItem", "position": 9, "name": "CCBot"}, {"@type": "ListItem", "position": 10, "name": "cohere-ai"}]}