ClaudeBot: What It Is, How to Verify It, What Blocking It Costs

ClaudeBot is Anthropic's web crawler. It reads public pages so Claude's models have current, citable content. You control it with one robots.txt token: ClaudeBot. But the user-agent string is easy to fake. As of July 2026, you confirm a real request by its IP, not by the string.

This page covers one crawler. You get its user-agent, a way to prove a request is genuine, the exact robots.txt directives, what its traffic looks like in real logs, and the cost no crawler doc states: what a block does to your Claude visibility. Want every AI crawler side by side? See the AI crawler list .

What is ClaudeBot?

ClaudeBot is the crawler Anthropic runs to gather public web content for Claude. In your logs it shows up under a user-agent that holds the token ClaudeBot. It is one of several AI crawler user agents you will meet in 2026. The others include GPTBot from OpenAI, PerplexityBot, and OAI-SearchBot.

Two older Anthropic tokens still turn up: anthropic-ai and Claude-Web. Google's own AI Overview for "block ai crawlers," captured July 9, 2026, still lists both. They sit in its sample block next to GPTBot, cohere-ai, CCBot, and Google-Extended. Treat them as legacy aliases. Treat ClaudeBot as the current token.

This page will not guess Anthropic's full roster. Anthropic splits bulk crawling from user-triggered fetching. The current list of tokens lives in its crawler docs. Pull it live before you write firm rules, because the roster changes.

User agent strings and IP verification

A ClaudeBot request carries a user-agent that holds the ClaudeBot token plus an Anthropic contact address. The wrapper around that token has changed between versions. So match on the token, not the full string. And never trust the string as proof. Anyone can send it. The July 8, 2026 audit of 34 sites worked by doing exactly that.

Verify ClaudeBot by IP, not the user-agent string. The string is trivial to spoof.

Here is the reference card, in one place:

Proof comes from the IP, and two checks work. First, Anthropic publishes the IP ranges its crawlers use. Check that the source IP sits inside that set. Pull the current list from Anthropic's docs, because ranges rotate. Second, if you run Cloudflare, let Cloudflare bot management do it. Its verified-bot list flags real traffic for you. A request that claims the name from an unlisted IP is a fake. You can drop it without touching real access.

robots.txt: allow or block ClaudeBot

One robots.txt block governs the crawler, and the whole decision is blocking vs allowing. To keep it out, disallow the ClaudeBot token. To invite it, allow it, or just leave it alone. Anthropic says the crawler respects robots.txt. But the rule is a request, not a fence. Google's AI Overview puts it plainly: non-compliant bots "may ignore" it.

To block it, disallow the token:

To allow it on purpose, though the default is already open:

To cover the two legacy tokens in the same file:

For real enforcement, not a polite ask, use Cloudflare. Its AI Crawl Control panel has a toggle you flip to Block. Its WAF can match the crawler at the edge. A user-agent-only rule catches lazy requests but is easy to spoof:

So pair it with Cloudflare's verified-bot category instead of trusting the string. This section is reference, not a campaign. Which way you set the toggle is the choice below.

Crawl behavior: what ClaudeBot looks like in logs

AI crawlers hit hard in 2026. One self-hosted site, tracked for 17 days to June 25, 2026, logged 4,523 human visits against 6,409 attack attempts. AI retrieval agents formed a separate class: GPTBot, ClaudeBot, and peers. So the crawler is active, but second-tier.

One monitoring network logged 300 to 500-plus AI-bot visits a day in October 2025. GPTBot took about 60%. Claude, Perplexity, and Google's bots split most of the rest. The crawl rate here is steady and well under GPTBot's share.

Two facts shape how you see it at all. First, client-side analytics miss it. One developer logged 1,011 AI crawler requests in 72 hours; Google Analytics reported zero (March 15, 2026). You find this traffic in raw server logs, nowhere else. Second, the scary crawl-rate stories are usually not Anthropic. The headline case was Meta. One owner watched Meta's crawler make 7.9 million requests in 30 days and burn 900-plus GB (r/webdev, April 6, 2026). That works out to about 263,000 requests a day, near 183 a minute, close to 30 GB daily. Reported ClaudeBot volumes run far below that spike. But the spike is why owners go looking, and then find this crawler on the list.

Is the load straining your origin? Throttle at the edge with Cloudflare rate-limiting. Support for a non-standard Crawl-delay is not promised, so test it first. First-party dated logs are the one input this reference still owes you.

ClaudeBot and your Claude visibility: the trade-off

Blocking the crawler carries a cost no doc prices for you. Content ClaudeBot never reads is content Claude cannot cite. If AI answers are a discovery channel for you, a Disallow: ClaudeBot line is also a "hide my brand from Claude" line. That is the trade-off, both camps standing.

Allow, then audit the edge for accidental blocks (the 27 percent trap). Or block on purpose and charge.

The risk is bigger than teams think, and it is usually an accident. A review of a few thousand US and UK sites found 27% blocking at least one major LLM crawler. The block usually lived at the CDN, WAF, or edge layer, not in robots.txt (r/aeo, February 18, 2026). The July 8 audit found 6 of 34 sites blocking ChatGPT outright, and "almost none of the owners knew." Stale rules do it quietly. A catch-all Disallow written before these bots existed now blocks GPTBot and the crawler by default (r/GenEngineOptimization, June 23, 2026). One nuance keeps this honest: "sites that block AI training crawlers mostly ignore the answer time bots" (Hacker News, July 7, 2026). Training-time crawling and answer-time retrieval are different levers. Check which one your rule stops.

The block camp is not wrong, and this page will not talk you out of it. The exchange is one-sided. AI crawlers "consume content, generate answers from it, but send little to no referral traffic back." A WordPress origin still pays server load per request (r/Wordpress, April 28, 2026). That is a fair reason to want publisher control. The middle path is pay-per-crawl. Cloudflare offers a model for it. One Caddy plugin even charges crawlers real USDC per request (Hacker News, February 27, 2026). It keeps the door open at a price instead of nailing it shut.

So the framework, no hedge. Does Claude visibility beat the crawl cost for you? Allow the crawler, then audit that you are not blocking it by accident at the edge. That is the 27% trap. Is your content the product, and the trade not worth it? Block on purpose, or charge for access. Either way, verify what you serve each bot. Do not let a stale robots.txt decide for you. Not sure which side you are on? Check whether AI crawlers can read your site . Once you decide to allow, use llms.txt as the explicit invite.

Частые вопросы