Case Study

I Audited 343 Websites for AI Search Visibility

Published July 21, 2026 · Updated October 4, 2026 · 9 min read · Reproducible study

I built GeoScore, a free tool that scores a website's AI-search readiness across 11 dimensions. The obvious next question is: what does the score distribution of the web actually look like?

This post is the answer, from a run you can reproduce: 343 real sites, audited with the same scoring code the browser audit uses.

Correction (2026-10-04). An earlier version of this post claimed 1,200 audited sites and reported a 3.4× citation lift. Both numbers were unverifiable — the database held 3 audit rows for a single domain, and a static audit cannot observe whether an AI engine cited anything. The study below replaces them. The original 1,200-site claim is withdrawn.

TL;DR

Median score: 41 out of 100. The typical site is not optimized for AI search, but it is not broken either.
44.4% rate Critical (under 40). 8.6% reach Good; nobody reaches Excellent.
19 of the 20 highest-scoring sites publish an llms.txt. Sites with one average 60.3; sites without average 38.0.
30.6% publish no robots.txt at all, and only 3.6% of the rest block any AI crawler.
My own scorer was wrong about this and overstated blocking by 25× — Finding 4 has the before-and-after.

How I ran the analysis

I wanted a sample that could be re-derived rather than a list I happened to have. So the rule is fixed and scripted:

Source: the GitHub Search API, repositories, sorted by stars descending.
Stratified: 12 fixed star bands, from stars:≥50000 down to stars:50..79, taking the top 100 repositories in each — 1,200 candidates.
Filtered: only repositories with a homepage, since a site is required. Hosting placeholders like *.github.io are excluded.
Deduplicated by registrable host, so one company counts once.

That left 457 unique sites. Each was audited with GeoScore's own scoring code, at concurrency 5 with a 15-second timeout; unreachable sites were retried once at 30 seconds.

Final sample: 324 scored, 133 unreachable (29.1%).

The failures are real and recorded, not discarded — mostly timeouts plus sites that refuse programmatic requests. Every number below comes from the 324 scored sites. That is comfortably above the 100-site floor I set for publishing anything at all.

A note on what this sample is: these are repositories with visible homepages, so it skews toward developer-facing products. It is not a representative sample of the web, and I am not going to pretend otherwise.

Two passes, one corpus. The first run scored 343 sites. After I fixed the robots.txt rule described in Finding 4, I re-ran the same 457 candidates with the corrected scorer; 324 came back. Every score in this post is from that second run, because it reflects what the tool actually does today. The first run's numbers survive only in Finding 4, as the before half of the comparison. The 19-site difference between runs is network conditions, not a change in method — 126 of the second run's failures were 15-second timeouts.

Finding 1: the median site scores 41

The distribution is bottom-heavy, though less so than the first run suggested:

Critical (0–39)144 sites — 44.4%
Basic (40–69)152 sites — 46.9%
Good (70–85)28 sites — 8.6%
Excellent (86+)0 sites — 0%

Median 41, mean 43.4, and the highest score in the entire sample was 80. None of these 324 sites reached the top band, and the lowest was 8.

The practical read: AI-search optimization is not a mature practice. Whatever the eventual benchmarks turn out to be, almost nobody has done this yet — which is good news if you have not started.

Finding 2: llms.txt is the clearest dividing line in the data

78 sites (24.1%) publish an llms.txt. The gap between them and everyone else is large:

With llms.txt (n=78)mean 60.3 · median 64
Without (n=246)mean 38.0 · median 38

Two ways to see the same split. Of the 67 sites scoring 60 or higher, 51 have an llms.txt. Among Critical sites, only 6.9% have one. The highest score reached by any site without an llms.txt was 72, and 19 of the top 20 have one.

What this is not. This is a correlation in a convenience sample, not a causal effect, and I am not claiming a "lift." Publishing an llms.txt is a signal that someone has done the thinking about what the site is and which pages matter — and that thinking shows up in other dimensions too. The honest statement is: llms.txt presence marks the sites that have been deliberately optimized, and those sites score about 22 points higher.

If you want the practice rather than the statistic, the ultimate llms.txt guide covers the spec and the common mistakes.

Finding 3: robots.txt is a very small problem

I expected this to be the headline problem. The data says it is close to a non-problem:

Scored sites324
Publishes a robots.txt225 — 69.4%
No robots.txt at all99 — 30.6%
Blocks at least one AI crawler (of the 225 that publish one)8 — 3.6%
Specifically blocks GPTBot (of the 225)5 — 2.2%

Eight sites out of 225. Every one of them does it deliberately, with an explicit named group:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Googlebot
Allow: /

The complete list: aka.ms, hackage.haskell.org and weibo.com block all 20 crawlers; catppuccin.com blocks 5; and hellogithub.com, rustdesk.com, unity.com, maxon.net block one each. Note the denominator: the 99 sites with no robots.txt are not counted as blocked. With no file, a crawler falls back to the permissive default, and folding them in would have produced a headline number roughly 25× too large.

One of these is worth a look, because it shows why "longest rule wins" is not the whole story. aka.ms publishes Disallow: / alongside Allow: /fwlink/p/. The Allow is the longer pattern, so a naive longest-match check reads it as "allowed" — but it does not cover the site root, so every crawler is in fact blocked. The question is not which rule is longest; it is which rule governs /.

Finding 4: my own scorer was wrong about robots.txt, and I fixed it

The version of this post published earlier reported that GeoScore's AI Crawlability dimension scored 323 of 343 sites (94.2%) at 0/12, and I set that column aside instead of quoting it. I have since fixed the dimension, and the fix changes the number so much that the old figure is worth explaining rather than just deleting.

The old check looked for an explicit User-agent: <bot> line containing Allow: /. That inverts RFC 9309: a file which never mentions an AI bot leaves it on the permissive default, and a site only lists a bot when it intends to restrict it. A perfectly ordinary file — User-agent: * with Disallow: /search?q=* — was scored as a total block. dev.to, fully open to GPTBot, scored 0/12.

The check now parses the file into groups, merges every group naming the bot, ignores the * group whenever a specific one exists, and asks a single concrete question: can this crawler fetch the site root? Re-fetching all 337 robots.txt files under both rules:

Old rule: sites with at least one crawler blocked228 — 67.7%
New rule: sites with at least one crawler blocked7 — 2.1%
Old rule: median AI Crawlability score0 / 12
New rule: median AI Crawlability score12 / 12

I verified the six sites whose verdict changed by reading their actual robots.txt, and the new answer is right in every case — catppuccin.com really does ban GPTBot, ChatGPT-User, Google-Extended, CCBot and PerplexityBot by name; rustdesk.com bans CCBot; unity.com bans Bytespider. The dimension is no longer a meaningful signal on typical sites, which is the correct outcome: a site that says nothing about AI crawlers is not penalised for it. If you scored your site before this fix, your AI Crawlability number went up, and the change is a correction, not a regression.

What this study cannot tell you

The previous version of this post made four claims I have removed rather than restate:

"3.4× citation lift for sites with a valid llms.txt"
"FAQ schema 6.1× more likely to be cited"
"HowTo schema 3.8× more likely to be cited"
"the top 3 fixes deliver ~80% of the score improvement"

Every one of those is a claim about whether an AI engine cited a site. A static crawler audit cannot observe citation — it fetches robots.txt and HTML, and citation happens inside a model. Measuring it would need a multi-engine citation crawl or an LLM simulation, and the simulation in my own product is an estimate rather than a measurement.

The 23-point llms.txt gap in Finding 2 is what I can actually evidence. It is a weaker claim than "3.4× lift," and it is the one I am willing to stand behind.

The three things the data supports

Ordered by how strongly the sample separates groups, not by how much score each one moves:

1. Publish an llms.txt (about an hour)

The strongest dividing line in the data: 24.1% of sites have one, and they average 22 points higher (mean 60.3 vs 38.0; median 64 vs 38). Write the Markdown summary at yourdomain.com/llms.txt following the spec, then reference it from robots.txt. Full guide here →

2. Check what your robots.txt actually says (five minutes)

3.6% of sites that publish a robots.txt block at least one AI crawler, and 30.6% publish no robots.txt at all — which is not the same thing, since a crawler with no policy is allowed by default. The one trap: do not add User-agent: * / Disallow: / while trying to fix it.

3. Fix the cheapest dimensions first (an afternoon)

Across the sample, Meta & Social sits at 73% of its maximum and Content Quality at 59% — most sites already do these. AI Crawlability (excluding the caveat above), Structured Data at 13%, and E-E-A-T at 25% are where the room is.

What I'd build differently

Ship the sampler before the blog post. The hard part of this study was not the analysis, it was making the sample reproducible. A rule someone else can re-run turns a claim into a result. I should have written tools/build-corpus.mjs first.
Separate measurement from marketing in the file layout. This data lives in data/ with a provenance note, and the blog post cites it. Nothing in the post is typed by hand.
Check the scorer against sites you already know the answer for. dev.to is not subtle, and the dimension that mis-scored it is the one I would have trusted least if I had not spot-checked two sites by hand.

Run your own audit

The 11-dimension browser audit is free and sign-in is optional. Drop in your URL, get a 0-100 score, a radar chart, and auto-generated fix files (llms.txt, robots.txt, JSON-LD) you can deploy immediately.

With a median of 41 across this sample, most sites have room.

Audit your site for free →