PyRank

Web Archiving Python Packages

Python packages with the GitHub topic web-archiving. Sorted by relevance, with stars and monthly downloads.
akamhy
waybackpy

Wayback Machine API interface & a command-line tool

5M 592 40
webrecorder
warcio

Streaming WARC/ARC library for fast web archive IO

774K 459 70
ArchiveBox
archivebox

🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

72K 28K 2K
webrecorder
pywb

Core Python Web Archiving Toolkit for replay and recording of web archives

10K 2K 238
cocrawler
cdx-toolkit

A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine

7K 211 34
adventurelands
storetle

Pip-installable wheel for streaming large document corpora (USPTO, arXiv, PubMed, web crawls). HTML-aware compression with random access, about 46% smaller than gzip WARC.

6K 0 0
webrecorder
cdxj-indexer

CDXJ Indexing of WARC/ARCs

5K 34 16
oduwsdl
ipwb

InterPlanetary Wayback: A distributed and persistent archive replay system using IPFS

4K 652 41
bellingcat
auto-archiver

Automatically archive links to videos, images, and social media content from Google Sheets (and more).

4K 1K 107
Own-Data-Privateer
hoardy-web

Passively capture, archive, and hoard your web browsing history, including the contents of the pages you visit, for later offline viewing, replay, mirroring, data scraping, and/or indexing. Your own personal private Wayback Machine that can also archive HTTP POST requests and responses, as well as most other HTTP-level data.

1K 132 10
Own-Data-Privateer
hoardy-web-sas

Passively capture, archive, and hoard your web browsing history, including the contents of the pages you visit, for later offline viewing, replay, mirroring, data scraping, and/or indexing. Your own personal private Wayback Machine that can also archive HTTP POST requests and responses, as well as most other HTTP-level data.

798 132 10
caltechlibrary
eprints2archives

Send records from an EPrints server to the Internet Archive and other web archives

473 4 0
internetarchive
fatcat-openapi-client

Perpetual Access To The Scholarly Record

359 121 18
GeiserX
wayback-archive

Download complete websites from the Wayback Machine with full asset preservation for offline viewing

325 19 7
internetarchive
scrapy-warcio

Scrapy WARC I/O

317 24 6
eliask
farchive

Local content-addressed archive with locator-scoped history for opaque bytes.

248 7 1
ArchiveBox
archivebox-likn

The self-hosted internet archive.

204 28K 2K
GeiserX
wayback-diff

Intelligent web page comparison tool with Wayback Machine support and visual regression testing

165 1 0
Florents-Tselai
warcdb

WarcDB: Web crawl data as SQLite databases

158 406 10
ikreymer
pywayback

Core Python Web Archiving Toolkit for replay and recording of web archives

1 2K 239