llms.txt adoption in the Tranco top 5,000
We fetched /llms.txt on every one of the 5,000 most-visited
domains on the web. Here is what actually exists.
As of 2026-07-11, 7.5% of the Tranco
top-5,000 domains serve a spec-valid llms.txt and 15.7% serve any file at /llms.txt. Those are shares of the full
5,000; 1,590 domains — mostly CDN, DNS and infrastructure
hosts with no web root — returned no HTTP response at all and are reported
separately as unreachable, so among the 3,410 that did
respond the rates are 11.0% valid and
23.0% present. Adoption skews to the
very top: 11.0% of the top 100 serve
a valid file versus 7.4% of ranks 101–5,000 — 1.5× more common among
the most-visited sites. 375 domains serve a valid file,
411 serve something malformed, and 2,624 respond but have nothing at /llms.txt.
llms.txt is a proposed standard: a Markdown file at the root of a domain that gives AI systems a curated map of a site's most useful pages. Publishing one is trivial. The open question is how many sites actually have one — so we measured it directly on the busiest slice of the web rather than estimating. Every number on this page comes from a single crawl; the raw JSON and CSV are linked below, licensed CC BY 4.0.
The malformed slice matters: 411 domains return a
200 at /llms.txt but serve an HTML page (usually a catch-all
route or a soft-404), not a Markdown file an AI system could parse. Counting
those as "adoption" would overstate it — so we separate them. Presence also
concentrates at the top: adoption in the top 100 runs well ahead of the long
tail, which fits llms.txt being, so far, a developer- and platform-led
convention rather than a mainstream one.
v1 crawled the Tranco top 1,000 (8.5% spec-valid, 12.7% of responders); v2 widens the lens to the top 5,000 (7.5% valid, 11.0% of responders). Both are same-day snapshots at different sample depths. From here the study is refreshed monthly: a Worker cron re-checks a fixed Tranco top-30 cohort and appends a row below — a smaller, constant cohort, so read each line against its own sample size.
| Run | Date | Cohort | Spec-valid | Of responders |
|---|---|---|---|---|
| V1 | 2026-07-11 | top 1,000 | 8.5% (85) | 12.7% |
| V2 | 2026-07-11 | top 5,000 | 7.5% (375) | 11.0% |
We attempted an adoption-by-category cut, but no category source cleanly covered the 5,000 — coverage ran out past roughly the top 250 (see Methodology → By category below). Rather than fabricate the missing ~98%, the by-category view is disclosed as unsourced and deferred to a future refresh.
- Source
- Tranco top-1M list 94Y92 (https://tranco-list.eu/list/94Y92), generated 2026-07-10 — 30-day rolling average, 11 Jun – 10 Jul 2026, five providers (Crux, Farsight, Majestic, Radar, Umbrella).
- Sample
- The top 5,000 ranked domains from that list, unmodified. List file SHA-256 9dc661485b406013…
- Method
- GET https://{domain}/llms.txt — concurrency 30, 5s per-request timeout, up to 2 redirects followed, response body capped at 256 KB, identifying User-Agent. /llms-full.txt fetched the same way.
- Classify
- present-valid = HTTP 200, non-empty, non-HTML, spec-shaped (H1 title or Markdown link sections). present-malformed = 200 but an HTML page or catch-all. absent = the host answered but served 404 / non-200 / soft-404 / empty body. unreachable = no HTTP response at all (timeout or connection error).
- Reachability
- An initial higher-concurrency pass left 3,263 domains in timeout/connection-error — many were false timeouts (fast sites that lost the race under load). Each was re-probed once, gently (concurrency 8, 8s). 1,673 recovered and were reclassified on their real response; the remaining 1,590 still returned nothing and are reported as unreachable — not silently counted as absent.
- Soft-404s
- A 200 that is an HTML "not found" page is content-sniffed and counted as absent (25 caught), not as a present file.
- Both framings
- 1,590 unreachable hosts (dead hosts and CDN/DNS/infra domains with no web root) are disclosed, not dropped. So we report presence two ways: out of the full 5,000 (7.5% valid / 15.7% any file), and out of the 3,410 domains that actually returned an HTTP response (11.0% valid / 23.0% any file).
- Date crawled
- 2026-07-11 (UTC). Re-runnable — the crawl + re-probe scripts and raw JSON ship with the dataset.
- Refresh
- Refreshed monthly. A Cloudflare Worker cron re-checks a fixed Tranco top-30 cohort on the 1st of each month and appends a snapshot to a D1 table, which this page reads for the running trend below. The 1,000- and 5,000-domain machine crawls are the seed baselines; the monthly pulse is a smaller fixed cohort, so each trend row is labelled with its sample size.
- By category
- By-category adoption was attempted via Cloudflare Radar content categories (GET /radar/ranking/domain/{domain} -> details_0.categories) with the account's Radar token. Coverage is confined to the head of the list: a probe returned categories for 64 of the top 100 and 99 of the top 250 domains, but 0 of a 10-domain sample spanning ranks 300-5,000 — roughly the top ~2% of the sample. The bulk Domain-Intelligence categorization endpoint returned an authentication error for the available token. With no source covering the other ~98% of the 5,000, the by-category cut is disclosed as unsourced and deferred rather than fabricated.
- Limitations
- Tranco ranks domains, not websites, so the sample includes CDN, DNS and API hosts that never serve a web root. Some servers bot-block or vary llms.txt by CDN edge, and a single fetch from one location can miss those or read as a false timeout (mitigated, not eliminated, by the gentle re-probe). Soft-404s are content-sniffed but detection is heuristic. The study measures presence + spec-shape only — not the quality of the links inside a valid file.