Warc Python Packages

fastwarc

A robust web archive analytics toolkit

1.2M 142 18

resiliparse

A robust web archive analytics toolkit

1.1M 142 18

warcio

Streaming WARC/ARC library for fast web archive IO

774K 459 70

archivebox

🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

72K 28K 2K

cdx-toolkit

A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine

7K 211 34

storetle

Pip-installable wheel for streaming large document corpora (USPTO, arXiv, PubMed, web crawls). HTML-aware compression with random access, about 46% smaller than gzip WARC.

6K 0 0

cdxj-indexer

CDXJ Indexing of WARC/ARCs

5K 34 16

forum-dl

Scrape posts and threads from forums, news aggregators, mail archives

4K 121 7

ipwb

InterPlanetary Wayback: A distributed and persistent archive replay system using IPFS

4K 652 41

scrapy-webarchive

A plugin for Scrapy that allows users to capture and export web archives in the WARC and WACZ formats during crawling.

816 9 1

internet-archive-extractor

Tool for extracting archived web sites from the Internet Archive saving as WARC files.

670 4 0

warc2zim

Command line tool to convert a file in the WARC format to a file in the ZIM format

643 85 7

cdxsummary

Summarize web archive capture index (CDX) files.

323 96 32

scrapy-warcio

Scrapy WARC I/O

317 24 6

metawarc

metawarc: a command-line tool for data extraction from WARC files (web archives)

303 34 2

cocrawler

A modern web crawler framework for Python

301 194 24

otmt

Tools for determining if web archive collecions are Off-Topic

267 9 3

archivebox-likn

The self-hosted internet archive.

204 28K 2K

warcdb

WarcDB: Web crawl data as SQLite databases

158 406 10