Web Archiving Python Packages

waybackpy

Wayback Machine API interface & a command-line tool

warcio

Streaming WARC/ARC library for fast web archive IO

archivebox

🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

pywb

Core Python Web Archiving Toolkit for replay and recording of web archives

cdxj-indexer

CDXJ Indexing of WARC/ARCs

spn-client

A hardened Python client for archive.org's availability API and Save Page Now (SPN2): process-wide rate pacing with a circuit breaker, dual-mode anonymous/S3-authenticated submission, and an explicit archive-outcome vocabulary.

ipwb

InterPlanetary Wayback: A distributed and persistent archive replay system using IPFS

cdx-toolkit

A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine

auto-archiver

Automatically archive links to videos, images, and social media content from Google Sheets (and more).

gnosis-markdown

Web scraper & crawler → clean Markdown with byte-level SHA-256 provenance + WARC archival for LLM/RAG knowledge bases

offprint

🖨️ Capture rendered web pages as self-contained HTML for offline use, with PDF and Markdown export

wayback-archive

Rescue whole websites from the Wayback Machine.

hoardy-web

Passively capture, archive, and hoard your web browsing history, including the contents of the pages you visit, for later offline viewing, replay, mirroring, data scraping, and/or indexing. Your own personal private Wayback Machine that can also archive HTTP POST requests and responses, as well as most other HTTP-level data.

kalika

Web archiver and crawler based on Heritrix and pywb.

storetle

Pip-installable wheel for streaming large document corpora (USPTO, arXiv, PubMed, web crawls). HTML-aware compression with random access, about 46% smaller than gzip WARC.

scrapy-warcio

Support for writing WARC files with Scrapy

eprints2archives

Send records from an EPrints server to the Internet Archive and other web archives

website-downloader

Website-downloader is a powerful and versatile Python script designed to download entire websites along with all their assets. This tool allows you to create a local copy of a website, including HTML pages, images, CSS, JavaScript files, and other resources. It is ideal for web archiving, offline browsing, and web development.

fatcat-openapi-client

Perpetual Access To The Scholarly Record

hoardy-web-sas

Passively capture, archive, and hoard your web browsing history, including the contents of the pages you visit, for later offline viewing, replay, mirroring, data scraping, and/or indexing. Your own personal private Wayback Machine that can also archive HTTP POST requests and responses, as well as most other HTTP-level data.

internet-historian

Quietly preserve the web things you love, forever — a patient Internet-Archive-only web archiver for macOS, with a Claude Code skill. Built to save ちいかわ (Chiikawa) pages.

farchive

Local content-addressed archive with observation history. Stores bytes by SHA-256, preserves locator state as contiguous spans, compresses with zstd and corpus-trained dictionaries. SQLite-backed.

warcdb

WarcDB: Web crawl data as SQLite databases.

archivebox-likn

🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...