Content Extraction Python Packages

web-researcher-mcp

The AI research assistant that cites real sources honestly — and searches the web. Your AI research assistant that cites real sources and stays honest. Works with Claude, Cursor, any MCP client.

contextractor

Crawl any website and extract clean, boilerplate-free main-content text as Markdown, plain text, JSON, HTML, or raw original HTML — ready for LLMs, RAG, and vector databases. Built on rs-trafilatura + Crawlee/Playwright. Ships as a CLI, npm library, Python package, and Apify Actor.

searchts

The missing layer between AI and the web. Open-source escalating web unlocker + read/search/transcribe/grab across web, GitHub, YouTube, Reddit, Twitter, LinkedIn, and RSS for AI agents.

argus-search

Multi-provider web search broker for AI agents. 14 providers, budget-aware routing, content extraction — one API.

article-extractor

Pure-Python article extraction library and HTTP API - Extract clean content from web pages as Markdown or HTML

netlens-mcp

MCP server for AI agents to read any web page in full — past the robots.txt and bot blocks that stop native fetch — as clean, ad-stripped Markdown. Plus web search.

pulpie

Pareto-optimal models for cleaning the web — fast, encoder-based main-content extraction from HTML.

extractnet

A fork of Dragnet that also extract author, headline, date, keywords from context, as well as built in metadata extraction all in one package

url2md4ai

Convert any URL into clean, token-efficient Markdown for LLMs. One thing, done well.

websearch-skill

Keyless, uv-native web search + read for AI agents: ddgs multi-engine search with de-correlated rank fusion, Trafilatura extraction to paginated Markdown, plus keyless arxiv and github search. One fenced agent face (web_search/web_fetch/web_open) and a FastMCP server. Installs via npx skills add, a Claude plugin, or uvx.

llamaindex-agent-toolbox

LlamaIndex integration for Agent Toolbox API — 13 tools for AI agents: search, extract, screenshot, weather, finance, news, translate, geoip, whois, dns, pdf-extract, and qr code generation

anything2md

Python package and CLI for converting documents to Markdown using Cloudflare Workers AI toMarkdown.

tikara

The metadata and text content extractor for almost every file type.

langchain-agent-toolbox

LangChain integration for Agent Toolbox API — 13 tools for AI agents: search, extract, screenshot, weather, finance, news, translate, geoip, whois, dns, pdf-extract, and qr code generation

fetchv2-mcp-server

A robust MCP server for fetching and extracting web content using Trafilatura. Optimized for AI agents with clean markdown output.

omniparse-rs

🦀 A Rust toolkit for detecting and extracting content from multiple file formats. CLI + Library API

webpeel

The web data layer for AI agents — fetch, search, crawl, extract, screenshot, and monitor the web with 50+ domain extractors and MCP.

jsuite

Tools for parsing and manipulating JATS XML documents.

docscraperforai

A powerful library to scrape documentation pages (or entire domains) into text, Markdown, or JSON files.

lurkers

Convenient API + CLI to fetch web content for agents

peduncle

content extraction from html