Do the Top Sites Actually Use robots.txt, Open Graph and llms.txt?
Every SEO tool assumes these files matter — but how many real sites bother? We scanned the Tranco top 1,000 for the signals a search engine (and now an AI crawler) looks for: robots.txt, a sitemap, AI-crawler rules, an llms.txt, Open Graph, a canonical, structured data. Of the 1,000, 554 served a browsable homepage — the rest are CDN and DNS infrastructure — and here is what those 554 actually ship.
The basics are table stakes; the rest isn’t
A meta description, a robots.txt and Open Graph tags are near-universal — four in five top sites have them, because sharing and crawling break without them. But it falls off fast after that: structured data (JSON-LD) is on fewer than half, and FAQPage schema is on just 2%. The “advanced SEO” layer that guides tools recommend is, in practice, mostly empty space.
Share of the 554 reachable top-1,000 sites carrying each signal. Amber = AI-crawler rules, green = llms.txt.
Who’s blocking the AI crawlers
Naming a crawler in robots.txt almost always means blocking it, and the pattern is clear: GPTBot is the most-targeted, with ClaudeBot and Google’s AI crawler close behind. Big publishers lead — the New York Times and Cloudflare both turn them away — and the rules to do it are only a few lines. This is the SEO question that didn’t exist three years ago.
Share of the 554 reachable sites naming each crawler in robots.txt (x-axis zoomed to 0–30%).
llms.txt is real, early, and very developer-shaped
The surprise was llms.txt — a plain-markdown file that tells AI models what a site is and where its key docs live. At 16% of the top sites it is already more common than FAQPage schema, and it barely existed 18 months ago. But look at who has one: Cloudflare, GitHub, Stripe, Shopify, WordPress, Adobe, Cisco — overwhelmingly developer-tool and infrastructure companies. It is spreading fastest where the people who read specs also write the website. Whether the rest of the web follows is the open question.
The practical takeaway for everyone else: the basics still win. A clean robots.txt with a sitemap line, real Open Graph tags and a meta description put you ahead of a surprising share of even the top 1,000 — and each is a two-minute fix. You can check your own site’s files with the free SEO & AI tools.
Reproduce this
One public site list, three fetches per domain, a short script — you can re-run the whole scan.
- Sample: the Tranco top 1,000 (a reproducible research list); 554 served a browsable HTML homepage, the rest are CDN/DNS infrastructure.
- Checks: robots.txt (+ Sitemap line, + AI-crawler names), homepage (Open Graph, meta description, canonical, JSON-LD, FAQPage), and a text/plain /llms.txt.
- Reported: adoption as a share of the 554 reachable sites, scanned as an identified research bot.
Two caveats: the top 1,000 skews commercial and tech-heavy, so adoption here runs ahead of the general web (llms.txt especially). And the web changes daily — the shape holds, the exact percentages will drift. Check any single site’s files with the SEO & AI tools.