webtelemetry.dev

Archived. Active from 21 September 2026 to 19 October 2026.

This project has been shut down. The crawler stopped on 19 October 2026, the servers are off, and nothing on this domain is updated any more. This page is a record of what it was.

What it was

webtelemetry.dev was a self-built web census. A crawler found websites on its own, visited their public pages and recorded what it could measure about each one. The site then showed the results as live feeds, per-site profiles and aggregate statistics. It was a solo project, built and run by one person on a single rented server.

What it looked like

Screenshots of the live site on the last day, 19 October 2026.

The webtelemetry.dev home page: a search box, and counters for websites tracked, launches found, share blocking AI crawlers and share with an llms.txt file — Home page with live counters.

The statistics page: websites tracked, new domains, launches found today and sites checked per minute — Statistics page: what the new web is built with.

The AI crawlers page comparing how often popular, older and new websites block AI crawlers — AI crawlers page: which AI crawlers new websites block.

How it found sites

New sites: from public Certificate Transparency logs, which record each newly issued TLS certificate.
Popular sites: from the Tranco list of top domains, revisited about every 30 days.
Registration dates: looked up through RDAP, so the "newly registered" views could tell new sites from old sites renewing a certificate.

What it measured

Security headers, TLS and certificate details, DNS and email authentication.
The technology a site was built with, detected from several thousand fingerprint rules.
Basic SEO signals and a 0 to 100 quality score that filtered out parked and placeholder pages.
Whether the site's robots.txt blocked each of fifteen AI crawlers, shown per crawler and per rank group.
Homepage screenshots for sites that passed a quality bar.

No page HTML was stored long term. Raw pages were held in a message queue for a few hours, and only the extracted measurements were kept in the database.

What the site offered

A live feed of newly discovered and newly registered sites.
A profile page for each crawled site.
Statistics pages, a screenshot gallery and a weekly report on AI crawler blocking.
An open daily dataset under CC BY 4.0, plus a small JSON API.

What it collected

Final figures, read from the database on 18 October 2026. The database was reset on 27 September, so these cover 27 September to 18 October. The crawler was fetching roughly 250 sites a minute at the end (358,375 sites in the last 24 hours).

4,985,209 domains profiled.
2,492,237 were alive, 1,339,066 dead, 663,752 blocked the crawler and 490,208 were parked pages.
870,180 changes to earlier-seen values were recorded.
Technology detection ran on a catalogue of 7,560 fingerprints.

Where the sites were

Top TLDs: .com (762,589), .xyz (635,925), .cc (286,384), .net (260,250), .online (234,626), .uk (234,295).
Top hosting countries: US (673,809), Canada (297,950), France (156,213), Germany (124,617). Sites behind anycast CDNs have no location and are not counted.
Top hosting networks: Cloudflare (1,538,529), Amazon (779,839), Hostinger (94,011), Google (51,015).
Top registrars: Onamae.com (436,786), Namecheap (405,908), Dynadot (298,747), GoDaddy (281,194), Spaceship (257,385).

What the alive sites looked like

2,423,703 of 2,492,237 (97%) answered over HTTPS with a TLS protocol recorded.
396,565 (16%) had an enforcing DMARC policy (reject or quarantine).
Most detected technologies: Cloudflare (1,294,671), Let's Encrypt (1,243,819), Google Fonts (775,201), PHP (586,449), jQuery (563,461), Nginx (396,310), WordPress (353,099), React (311,542).

AI crawlers

Of the 1,916,555 alive sites that had a robots.txt, 87,564 (4.6%) blocked at least one of the fifteen AI crawlers the project tracked. 62,405 (3.3%) blocked GPTBot, 62,457 (3.3%) blocked CCBot and 48,655 (2.5%) blocked ClaudeBot. A daily series of these figures ran from 3 to 18 October 2026.

The crawler

The crawler identified itself in its user agent and published a description at /bot. It used only ports 80 and 443, read robots.txt and honoured Crawl-delay. It treated a server error on robots.txt as "disallow" and backed off when a server answered 429 or 503.

Tech stack

Language: TypeScript on Node.js 22 for every service.
Pipeline: Apache Kafka (KRaft mode) connected the stages: discovery, scheduler, fetcher, parser and the database writer. They ran as roles inside one Node process and talked only through Kafka topics. Raw page HTML lived in Kafka for two to three hours and was never stored in the database.
Crawler: kafkajs, cheerio for HTML parsing, robots-parser, tldts for domain handling, node-forge for certificates, pino for logs. Parsing of hostile pages ran in worker threads with time and memory limits.
Database: PostgreSQL 17 on the same server, with LISTEN/NOTIFY pushing live updates to the site.
Website: Astro server-rendered with the Node adapter, sharp for images, live updates over Server-Sent Events. It read the database through a read-only role.
Screenshots: Playwright with headless Chromium in its own container, plus nsfwjs (TensorFlow.js) to skip adult content.
Location data: the DB-IP database, read with maxmind.
Admin panel: Express with password and TOTP two-factor login, charts, live tuning settings and a Kafka view (AKHQ).
Hosting: one Contabo VPS (Ubuntu 24.04), everything in Docker Compose, Caddy as the web server, Unbound as the local DNS resolver, Cloudflare in front for DNS and caching. Nightly database dumps and a weekly restore test ran from cron.

Timeline

21 Sep 2026: first commit; Kafka and Postgres set up on the VPS, crawler running within the day.
22 Sep: product rework. Registration dates from RDAP, a quality score and parked-page detection, so the "new sites" feed stopped being full of old sites renewing certificates.
27 Sep: trimmed to a small public site (launches, stats, gallery, AI rules, data) and the open daily dataset started. The database was reset; the figures above start here.
28 Sep: server upgraded to 8 vCPU and 24 GB RAM.
2 to 3 Oct: full security and reliability audit, fixes deployed; a consent-based analytics banner and the per-crawler AI pages and weekly report added. Daily AI crawler snapshots began.
12 Oct: first frozen weekly AI crawler report.
19 Oct: shut down.

Why it ended

It was a solo learning project, and revenue was never the goal. The aim was to learn how to build and run a real data pipeline: Kafka, large-scale crawling and data processing, Postgres, and running everything on a rented VPS at Contabo. That goal was met. Running it longer would have added cost and upkeep without teaching much more, so it was shut down. This page records what it did.

Limits of the data

Not a random sample. Sites came from Certificate Transparency logs (skewed to sites that recently got a certificate, and to the free-certificate and cheap-TLD end of the web) and from the Tranco top list. The totals do not describe "the web" as a whole.
"Alive" means a usable page was fetched. A site can be up and still be counted blocked, dead or parked if it refused the crawler, timed out or served a placeholder page.
Blocking is undercounted. Sites that blocked or rate-limited the crawler have no robots.txt reading, so they are missing from the AI crawler figures.
Location is incomplete. Sites behind anycast CDNs such as Cloudflare have no meaningful location and were left blank; about 46% of live sites had none when this was measured on 27 September.
Hosting and registrar come from IP and RDAP lookups. Several country-code registries publish no RDAP, so registration dates are missing for them.
Technology detection is pattern-based. Rules that need a real browser were skipped, so counts are a lower bound.
The period is three weeks. Change counts and trends are short; most sites were seen only once or twice.

Lessons from building it

A message queue hid a stalled stage. The launch rate fell to zero while discovery looked healthy. Three bugs combined: a response stream that never finished, a page that exceeded the queue's message size limit, and a consumer group stuck on the broker side. All fixed; throughput went from flat to about 500 sites a minute.
Storing raw HTML in Kafka filled the disk. 24 GB in under a day, from default seven-day retention. Cut to two hours with size caps.
"New certificate" is not "new site". Most entries in the first launches feed were established sites renewing. Registration dates fixed that.
Hostile pages are a real risk. Crafted HTML and regexes can freeze a parser, so parsing moved to bounded workers. Private-address filtering and DNS rebinding checks mattered for the screenshot browser too.
Default Postgres settings were far too small for a 12 GB host and were retuned after the first week.

Licences

Code: the project bundled GPL-3.0 fingerprint rules (see credits below).
Data: while it ran, the open dataset and AI crawler figures were published under CC BY 4.0. Rank data derived from the Tranco list was not included in the dataset because parts of that list are non-commercial.

Credits

Technology detection used the fingerprint rules of HTTP Archive's fork of Wappalyzer and of enthec/webappanalyzer, both licensed GPL-3.0. Site rankings came from the Tranco list. Location data came from DB-IP.