Measurement

The Search Box You Can’t Use

The web has a standard for asking a site to find its own page. It works on 1.3% of domains — and the reason is written in their robots.txt.

199,987 domains probed · July 2026

Here is a problem every search engine has, and almost nobody outside one thinks about: you want one page from a site. Not the site. One page — the one that answers the question somebody just asked.

The brute-force answer is to crawl the whole domain and keep what you need. That’s expensive for you and rude to them, and it’s most of why crawlers have the reputation they do. The polite answer is to use the map the site already published.

There are three such maps. A sitemap lists the URLs a site wants indexed. A feed lists what’s new. And OpenSearch — a standard from 2005, supported by every major browser — lets a site advertise its own search endpoint, so you can hand it a query and let it find the page for you.

OpenSearch is obviously the best of the three. No crawling at all. The site knows its own content better than any external index ever will. You ask, it answers.

So: how often does it actually work? I couldn’t find anyone who’d published the number from the crawler’s side, so I measured it.

The result

1.3%. The mechanism designed for precisely this purpose, supported by every browser for twenty years, is unusable on ninety-nine percent of the web.

Why

It isn’t that sites don’t publish it. Plenty do — the little <link rel="search"> tag is sitting right there in the head of the document, pointing at a description file that tells you exactly how to query them.

The problem is what’s in the other file. They publish the description, and then in robots.txt they Disallow: the endpoint it points at.

Which is, to be fair, completely reasonable. A search endpoint is an infinite URL space. Every query string is a new page. Let a naive crawler loose on one and it will happily generate requests until somebody’s pager goes off. Blocking it is basic hygiene, and site operators have been doing it since long before anyone thought about AI crawlers.

The clearest case is Wikipedia, which states the whole situation in two lines. It Disallow:s /w/ — where its search lives — while explicitly Allow:ing its sitemap.

It is telling you, in machine-readable terms: don’t ask me, read the map.

Mozilla’s developer docs are the interesting exception — MDN publishes OpenSearch and permits it. But its results render client-side, so a crawler that fetches the URL gets an empty shell. Allowed, and still useless.

The part that only shows up if you obey the rules

Here’s what makes this measurement worth publishing rather than just interesting: you can only see it if robots compliance is switched on.

A probe that ignores Disallow: would report OpenSearch working on far more domains, because the description file is right there and the endpoint usually responds. It would be measuring a capability no compliant crawler can use. The 1.3% isn’t a fact about what exists — it’s a fact about what you’re permitted to use.

Which means the honest conclusion for anyone building this: sitemaps and feeds are the web’s real index. Not because they’re better designed, but because they’re the ones publishers actually let you read. Design your crawler around the map, not the search box.

A second thing, which surprised me more

I ran the same probe against different slices of a domain ranking, expecting reachability to be roughly flat. It isn’t.

The top of the ranking is the least reachable part of the web. Which makes sense the moment you look at what’s actually up there: googleapis.com, gstatic.com, tag managers, font hosts, certificate authorities, CDN endpoints.

These domains outrank almost every newspaper on earth. They have no sitemap, no feed, no search, and nothing whatsoever to read — because they aren’t sites. They’re plumbing. A PageRank-style score measures how much of the web points at you, and everything points at the plumbing.

So any pipeline that takes “top N domains by authority” as a proxy for “the best content on the web” is mostly ingesting infrastructure. And reachability turns out to be a decent content filter in its own right — a domain with no sitemap, no feed and no search usually isn’t a site anyone reads.

How it was measured

At most three cheap GETs per domain:

/robots.txt   → Sitemap: directives (fallback: probe /sitemap.xml,
                accept only if the body really contains <urlset>)
/             → <link rel=search>    (OpenSearch description)
                <link rel=alternate> (RSS / Atom autodiscovery)

Robots rules honoured throughout, including for the OpenSearch endpoint — which, as above, is the whole point. Honest bot user-agent, 8-second timeout, concurrency 24, one vantage point. No content stored; only whether a mechanism exists.