Warc Python Packages

fastwarc

A robust web archive analytics toolkit

resiliparse

A robust web archive analytics toolkit

warcio

Streaming WARC/ARC library for fast web archive IO

archivebox

🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

forum-dl

Scrape posts, threads from forums, news aggregators, mail archives, export to JSONL, mailbox, WARC

cdxj-indexer

CDXJ Indexing of WARC/ARCs

ipwb

InterPlanetary Wayback: A distributed and persistent archive replay system using IPFS

cdx-toolkit

A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine

gnosis-markdown

Web scraper & crawler → clean Markdown with byte-level SHA-256 provenance + WARC archival for LLM/RAG knowledge bases

metawarc

metawarc: a command-line tool for metadata extraction from files from WARC (Web ARChive)

internet-archive-extractor

Tool for querying the Internet Archive CDX API, downloading resources and packaging them as WARC files.

warc2zim

Command line tool to convert a file in the WARC format to a file in the ZIM format

storetle

Pip-installable wheel for streaming large document corpora (USPTO, arXiv, PubMed, web crawls). HTML-aware compression with random access, about 46% smaller than gzip WARC.

scrapy-warcio

Support for writing WARC files with Scrapy

scrapy-webarchive

A plugin for Scrapy that allows users to capture and export web archives in the WARC and WACZ formats during crawling.

cdxsummary

Summarize web archive capture index (CDX) files.

otmt

This system evaluates a collection of mementos (archived web pages) to determine which are off topic. The collection can be part of an Archive-It collection, a single TimeMap, or stored in a WARC file.

cocrawler

CoCrawler is a versatile web crawler built using modern tools and concurrency.

warcdb

WarcDB: Web crawl data as SQLite databases.

archivebox-likn

🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...