Content Extraction Python Packages

web-researcher-mcp

Give your AI assistant real web search, full-page reading, and multi-source research with citations that are never fabricated.

searchts

A keyless web reader for AI agents. When a fetch hits a bot wall or a page that only renders in a browser, searchts tries a plain fetch, then a relay, then a real browser, and says plainly when it can't. MCP server, CLI, and Python library.

chromerag

HTML to RAG-ready Markdown. A 150 KB learned filter removes nav, footers, cookie banners, related links and comments; beats Trafilatura on held-out WCXB (F1 0.902 vs 0.860). CPU-only, ~36 ms/page.

argus-search

Multi-provider web search broker for AI agents. 14 providers, budget-aware routing, content extraction — one API.

extractnet

A fork of Dragnet that also extract author, headline, date, keywords from context, as well as built in metadata extraction all in one package

htmlsift

Main-content extraction for web pages: hand it a raw HTML string and get back the main content

article-extractor

Pure-Python article extraction library and HTTP API - Extract clean content from web pages as Markdown or HTML

websearch-skill

Keyless, uv-native web search + read for AI agents: ddgs multi-engine search with de-correlated rank fusion, Trafilatura extraction to paginated Markdown, keyless arxiv and github search. Self-hosted SearXNG without Docker, an optional Tor layer for .onion search and fetch, NordVPN/SOCKS5 egress, one-call init, and a doctor.

fetchv2-mcp-server

A robust MCP server for fetching and extracting web content using Trafilatura. Optimized for AI agents with clean markdown output.

netlens-mcp

MCP server for AI agents to read any web page in full — past the robots.txt and bot blocks that stop native fetch — as clean, ad-stripped Markdown. Plus web search.

markmaton

Lightweight HTML-to-Markdown parser for AI agent workflows — Python CLI and API around a fast Go engine, available on PyPI.

htomd

Extract main content and metadata from HTML as Markdown. Available in Python, TypeScript, Go, and Rust, with native APIs and CLIs.

langchain-agent-toolbox

LangChain integration for Agent Toolbox API — 13 tools for AI agents: search, extract, screenshot, weather, finance, news, translate, geoip, whois, dns, pdf-extract, and qr code generation

llamaindex-agent-toolbox

LlamaIndex integration for Agent Toolbox API — 13 tools for AI agents: search, extract, screenshot, weather, finance, news, translate, geoip, whois, dns, pdf-extract, and qr code generation

tikara

The metadata and text content extractor for almost every file type.

omniparse-rs

🦀 A Rust toolkit for detecting and extracting content from multiple file formats. CLI + Library API

pulpie

Pareto-optimal models for cleaning the web — fast, encoder-based main-content extraction from HTML.

anything2md

Python package and CLI for converting documents to Markdown using Cloudflare Workers AI toMarkdown.

url2md4ai

Convert any URL into clean, token-efficient Markdown for LLMs. CLI, Python API, MCP server.

public-source-extractor

CLI for turning one public web page into reviewable Markdown or JSON for AI research (experimental provider).

wanish

Summarize web articles: extract clean content, title, image, language, and generate text summaries

jsuite

Tools for parsing and manipulating JATS XML documents.

trafipy

Fast, accurate web content extraction in Rust. ML page-type classification, per-type extraction, confidence scoring. F1=0.966 on ScrapingHub (#1), F1=0.859 across 2,008 annotated pages (1,497 development + 511 held-out test

reddipy

Reddit data extraction toolkit for Python. Search posts, scrape comments, browse subreddits, and track user activity with built-in pagination, proxy rotation, and export to JSON/CSV.