Trafilatura Python Packages

contextractor

Crawl any website and extract clean, boilerplate-free main-content text as Markdown, plain text, JSON, HTML, or raw original HTML — ready for LLMs, RAG, and vector databases. Built on rs-trafilatura + Crawlee/Playwright. Ships as a CLI, npm library, Python package, and Apify Actor.

harnais-web-extractor

Two-stage web article extraction (trafilatura + Playwright fallback) with YouTube transcript support. CLI + library, JSON or Markdown output. No framework.

url2md4ai

Convert any URL into clean, token-efficient Markdown for LLMs. One thing, done well.

mineru-html

MinerU-HTML: An SLM-powered HTML main content extractor that outputs clean HTML bodies. Perfect for Deep Research Agents, RAG applications, and training data generation.

websearch-skill

Keyless, uv-native web search + read for AI agents: ddgs multi-engine search with de-correlated rank fusion, Trafilatura extraction to paginated Markdown, plus keyless arxiv and github search. One fenced agent face (web_search/web_fetch/web_open) and a FastMCP server. Installs via npx skills add, a Claude plugin, or uvx.

ingestforge

Auditable AI content ingestion: safe fetch, extraction, evidence gates, provenance ledgers, RAG datasets, and configurable provider/API exports.

lurkers

Convenient API + CLI to fetch web content for agents

pipescraper

A pipe-based news article scraping and metadata extraction library