Common Crawl Index Server (CDX/CDXJ)
Common Crawl Index Server (CDX/CDXJ)
📌 A free public CDX/CDXJ query service at index.commoncrawl.org that maps URLs to Common Crawl WARC captures. It turns petabytes of open web archives into searchable, selectively fetchable data for researchers and builders.
🔎 What it is
Public Index Server (index.commoncrawl.org) exposing a pywb-based CDX/CDXJ capture index. Pick a crawl snapshot, query a URL or pattern, get metadata pointing to the exact WARC record.
🌐 Why it matters
Common Crawl holds hundreds of billions of pages on AWS Open Data. Without an index, finding one site is impractical; the index enables NLP datasets, link graphs, historical studies, and SEO research without full-crawl downloads.
🔌 How to access
Free web UI and HTTP API (curl, cdx_toolkit). HTTPS required. Public endpoint is rate-limited; raw indexes live at s3://commoncrawl/cc-index/collections/ for self-hosting or offline use.
📊 Bulk alternative
For large domains or aggregations, use the separate URL Index (Parquet columnar) or download index files—do not bulk-query the rate-limited public CDX API.
⚙️ Stack & source
Built on the pywb web-archive index stack (Common Crawl Index Server). Same CDX Server API pattern as Wayback-style archives; code at github.com/commoncrawl/cc-index-server.
Key facts
| Fact | Value |
|---|---|
| Primary URL | https://index.commoncrawl.org/ |
| Index type | CDX / CDXJ capture (URL) index for WARC archives |
| Raw index location | s3://commoncrawl/cc-index/collections/ (also data.commoncrawl.org) |
| Software | pywb-based cc-index-server (GitHub: commoncrawl/cc-index-server) |
| Access model | Free UI + HTTP API; HTTPS required; public endpoint rate-limited |
| Bulk option | URL Index (Apache Parquet columnar) for filtering & aggregation |
Details
The Common Crawl Index Server is a free public query service that lets anyone look up which pages appear in Common Crawl’s open web archives. You select a crawl snapshot, enter a URL or pattern, and receive CDX-style metadata pointing to the WARC record that holds the captured page. The service is a deployment of the pywb web-archive index stack and follows the same CDX Server API used by Wayback-style archives.
Common Crawl stores petabytes of monthly crawls on AWS Open Data. The index makes that corpus practical to search and fetch selectively—for NLP training, link graphs, historical site studies, SEO research, and more—without downloading entire crawl archives. Index files themselves are free on S3, so heavy users can run their own server or query offline instead of hitting the public endpoint.
A separate URL Index (formerly Columnar Index) in Apache Parquet supports bulk filtering and analytics. The live Index Server is best for targeted URL lookups; for large domains or aggregations, Common Crawl recommends the columnar index or downloaded index files rather than bulk-querying the rate-limited public CDX API.