Checker Finds #1: robots.txt is never the problem. Schema is.

Across the 13 domains our probes have logged so far, every single one — 13 of 13 — passes the robots.txt check for AI crawlers. The failures live one layer up: 7 of 13 fail or half-fail JSON-LD schema, and 6 of 13 stumble on llms.txt. The popular fear points the wrong way.

Checker Finds is a recurring note: aggregated patterns from logged runs of our free checker and the hourly Pulse board . Small sample, honestly labeled — this batch is 13 domains, most of them famous tech sites, so treat it as a floor, not a census.

What 13 logged runs actually show

The per-signal tally from the logs:

• GPTBot / ClaudeBot access (robots.txt): 13 PASS, 0 failures. Nobody in this set blocks AI crawlers where you'd look for it.
• AI-crawler reachability: 12 PASS, 1 FAIL — and the one failure is the openai.com 403 we covered yesterday : the block lived at the WAF, invisible in robots.txt.
• JSON-LD schema: 6 PASS, 1 WARN, 6 FAIL — the worst signal on the board.
• llms.txt: 7 PASS, 2 WARN, 4 FAIL.
• Answer-first structure: 9 PASS, 3 WARN, 1 FAIL.

Why this matters for your site

The industry conversation obsesses over "should I allow GPTBot" — a check that, in our logs, everyone already passes. Meanwhile more than half the set can't tell an engine what kind of entity they are, because the homepage carries no parseable schema. An engine that can't classify you struggles to cite you, however open your robots.txt is.

The order of operations writes itself: confirm reachability beyond robots.txt (the WAF is the real gate — the AI crawler playbook shows how to verify), then ship entity schema, then a curated llms.txt . That's also exactly the order our audits prioritize fixes in.

The sample will grow — every checker run and hourly Pulse sweep adds rows, and future Checker Finds will re-cut these numbers on bigger n. Run your own domain and you're in the next batch (anonymously, score-level only).