How this works, and its limits

webtelemetry.dev is an automated census of websites as they appear. This page describes every step from discovery to what you see, and, just as important, where the numbers are incomplete or skewed. Figures below are live for the dataset collected since 22 September 2026.

1. Discovery

Every TLS certificate a public authority issues is written to Certificate Transparency (CT) logs, append-only public records browsers require. The crawler reads new entries from a small number of these logs (currently three, all operated by Google) and extracts each registrable domain (shop.example.co.uk becomesexample.co.uk).

A second, lower-priority source is links: domains linked from the homepages we crawl. Link-found sites are mostly established sites rather than new ones; they are fetched only when the CT queue has spare capacity.

2. Fetching

Each domain is visited once by WebtelemetryBot: it reads robots.txt first and honours it, fetches only the homepage (plus up to three of the site's own scripts), waits at least two seconds between requests to the same host, and never connects to private or internal addresses. Live sites are revisited every 7 days. Details and opt-out: /bot.

3. Profiling

Tech stack

Passive fingerprints (104 technologies) matched against HTML, headers, cookies and script URLs. Only platforms, CMSs, shop systems, frameworks and server languages count as built with. CDNs, web servers, analytics and tag managers are listed separately, because running behind Cloudflare or loading Google Tag Manager says nothing about how a site is built.

Kind of site

A rule-based classifier reads the title, description, structured data (JSON-LD), platform and TLD, in English and seven other languages, and picks one of 14 kinds (shop, SaaS, blog, local business, …). It only labels a site when the signals agree; everything else stays unlabelled rather than guessed.

Hosting and location

The IP address the crawler connected to, looked up in the free DB-IP Lite databases: network owner (ASN), country and approximate city. This locates the server, not the people behind the site.

Quality score

0 to 100, from whether the page has a real title and description, content, a detected platform, structured data and basic setup. Parked domains, default server pages and empty "coming soon" shells score low.

Screenshots

One homepage screenshot per newly registered site that passes the quality bar, taken with a headless browser under the same rules as the crawler.

4. What counts as "new"

Most certificates are renewals of existing sites, so a certificate alone does not mean a site is new. TheNewly registered views use the domain's registration date from its registry, looked up overRDAP, and include only domains registered in the last 30 days.

5. What is filtered out

• Parked domains, placeholders and dead sites never appear in feeds or browse pages. The public feed shows only live sites with a quality score of 40 or more.
• Adult, drug, weapons, piracy and scam sites are removed entirely: their data is deleted and the domain is never fetched again. Three checks decide this: rules on the page's own text and rating tags, Cloudflare's public family DNS filter, and an image classifier run on the screenshot. So far 547 domains have been removed this way.Gambling sites are not removed.
• Security checks (headers, email authentication) are collected for research but not published.

6. Known limits and biases

• Not every new site is seen. Only a few CT logs are read, and sites without a publicly trusted certificate never appear. The census is a large sample, not a complete list.
• Many sites have no detected stack. 41% of live sites (38,476 of 94,756) show no platform or framework. Detection is passive and custom-built or well-hidden stacks leave nothing to match. These are shown as "not detected", never guessed.
• Categories cover a minority of sites. 37% of live sites get a kind; the rest have too little text or conflicting signals. Rules favour precision over coverage, and some mistakes remain.
• Registration dates are missing for some TLDs. RDAP returned a date for 76% of the domains looked up. Registries such as .de, .io and .co do not publish them, so those domains can never appear as "newly registered", and name and TLD statistics lean towards .com and new gTLDs.
• Hosting country is skewed. Canada hosts 34% of live sites in our data. A large share of cloud and platform address ranges (AWS, Shopify and others) is geolocated there by DB-IP Lite, so read country as "where the provider's addresses are registered", not where sites or owners are.
• A young dataset. Week-over-week comparisons need two full weeks of data and trend lines get more reliable with time. Pages say so where it matters.
• Content removal is automatic. Cloudflare's family filter has some false positives, so a small number of harmless sites are removed along with the ones that should be. Site owners can write tosupport@webtelemetry.dev.

Current dataset

Data is available as JSON through the API. Found something wrong? support@webtelemetry.dev.