Common Crawl Data Stored on a Hugging Face Bucket

Common Crawl has mirrored portions of its 2026 crawl archives (April-June) on a Hugging Face Storage Bucket, keeping the same S3-style directory layout and enabling access via the hf CLI, hf-mount, S3-compatible APIs, and fsspec’s hf:// protocol. The bucket offers CDN pre-warming in several regions and allows browsing, streaming, or mounting so tools can read WARC files on demand without downloading entire archives. Examples show how to use warcio with fsspec to iterate WARC records and how to use DuckDB to inspect the URL index Parquet files over HTTP; a sample query counts pages per .ru registered domain and returns high-frequency hosts (e.g., yandex.ru).

Measured performance indicates S3→S3 (us-east-1) achieved the highest throughput (~3,500 records/s), HF→HF reached ~1,500 records/s, and HF→edge clients were much slower (<100 records/s) when issuing many small range requests due to rate limiting, so users should benchmark for their workloads. The release includes ready-to-run Jupyter notebooks (cc-index-hf.ipynb, s3-hf.ipynb, warcio-hf.ipynb) and GitHub examples. Note archived content can be truncated by fetch-size limits (raised from 1 MiB to 5 MiB in March 2025), and feedback is solicited via Discord and a Google Group.