Trafilatura Python Packages

chromerag

HTML to RAG-ready Markdown. A 150 KB learned filter removes nav, footers, cookie banners, related links and comments; beats Trafilatura on held-out WCXB (F1 0.902 vs 0.860). CPU-only, ~36 ms/page.

satchel-reader

A local-first reading queue: saves the real article text, not just the link, and makes it genuinely offline-searchable.

sifthound

Open-source, self-hosted Tavily alternative: web search, extract, crawl and map API for AI agents, with an MCP server

websearch-skill

Keyless, uv-native web search + read for AI agents: ddgs multi-engine search with de-correlated rank fusion, Trafilatura extraction to paginated Markdown, keyless arxiv and github search. Self-hosted SearXNG without Docker, an optional Tor layer for .onion search and fetch, NordVPN/SOCKS5 egress, one-call init, and a doctor.

siftdog

Open-source, self-hosted Tavily alternative: web search, extract, crawl and map API for AI agents, with an MCP server

mineru-html

MinerU-HTML: An SLM-powered HTML main content extractor that outputs clean HTML bodies. Perfect for Deep Research Agents, RAG applications, and training data generation.

url2md4ai

Convert any URL into clean, token-efficient Markdown for LLMs. CLI, Python API, MCP server.

harnais-web-extractor

Two-stage web article extraction (trafilatura + Playwright fallback) with YouTube transcript support. CLI + library, JSON or Markdown output. No framework.

trafipy

Fast, accurate web content extraction in Rust. ML page-type classification, per-type extraction, confidence scoring. F1=0.966 on ScrapingHub (#1), F1=0.859 across 2,008 annotated pages (1,497 development + 511 held-out test

pipescraper

A pipe-based news article scraping and metadata extraction library for Python

ingestforge

Safe, config-driven Python web ingestion pipeline with extraction, evidence-gated AI generation, provenance ledger, RAG chunks, data cards, and multi-provider exports.

lurkers

Convenient API + CLI to fetch web content (HTML, YouTube, Twitter, RSS) for agents