Open dataset
Every site in the census, one file per day: what it is built with, which network serves it, who runs its DNS and mail, how it handles AI crawlers. Free to use for anything, including commercially, as long as you credit webtelemetry.dev.
4b0c614a9ee24fbc049f617011bbc45d9c3b11381f33be388eaec054b68eee11Quick start
curl -O https://webtelemetry.dev/datasets/webtelemetry-latest.csv.gz
# DuckDB reads it as it is, gzip and all
duckdb -c "select technologies, count(*) from 'webtelemetry-latest.csv.gz'
where status = 'alive' group by 1 order by 2 desc limit 20"
# pandas
import pandas as pd
df = pd.read_csv("webtelemetry-latest.csv.gz")The link above always points to the newest complete file. For a fixed snapshot, use a dated file from the list below; those never change once written. A new file is published every day after 01:30 UTC, and the last 14 days are kept.
Columns
| Column | Meaning | Example |
|---|---|---|
domain | Registrable domain | example.com |
status | alive, parked, dead or blocked (robots.txt turned us away) | alive |
first_seen | Day we first profiled it | 2026-09-27 |
last_crawled | Day of the most recent visit | 2026-09-27 |
registered_on | Registration date from the registry (RDAP); empty where the registry does not publish one | 2026-09-24 |
registrar | Registrar named in the RDAP record | NameCheap, Inc. |
tld | Everything after the first label | co.uk |
category | Kind of site, classified automatically and approximately | shop |
technologies | Detected technologies, separated by | | Shopify|Cloudflare |
hosting_asn | Network (autonomous system) the site is served from | 13335 |
hosting_org | Owner of that network | Cloudflare, Inc. |
country | Server country, two letters; empty behind anycast CDNs, which have no single location | DE |
dns_provider | Who answers for the domain, from its NS records | Cloudflare |
email_host | Who handles its mail, from its MX records | Google Workspace |
has_spf | Publishes an SPF record | true |
dmarc_policy | DMARC policy: none, quarantine or reject | reject |
has_ipv6 | Publishes an AAAA record | false |
has_robots_txt | Serves a robots.txt | true |
ai_crawlers_blocked | AI crawlers disallowed from the site root, separated by | | GPTBot|CCBot |
has_llms_txt | Serves an llms.txt | false |
tls_issuer | Issuer of the TLS certificate | Let's Encrypt |
tranco_rank | Rank in the Tranco top million, if listed | 4821 |
Empty means unknown, not false. How each field is measured, and where it is incomplete or skewed, is on the methodology page.
What is not in it
- Security findings. The crawler records some to keep its own checks honest; they are never published.
- IP addresses and coordinates. The network and country say where a site is served from; nothing finer.
- Page text. Titles and descriptions are the sites' own words, so they stay on the profile pages.
- Removed and flagged sites. A site whose owner asked to be removed, or held back for its content, is not in the census, so it drops out of the next day's file. Files already downloaded cannot be recalled.
Licence and credit
The dataset is published under Creative Commons Attribution 4.0.
Credit it as webtelemetry.dev open dataset with a link to this page. Two fields carry their sources' own terms:
country,hosting_asnandhosting_orgcome from IP Geolocation by DB-IP, also CC BY 4.0.tranco_rankcomes from the Tranco list; its authors ask that research using it cite Le Pochat et al., "Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation", NDSS 2019.
Questions, or a use for the data we should know about: support@webtelemetry.dev. For individual sites or live feeds, see the API and alerts.