Corpus Tools Python Packages

trafilatura

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

simplemma

Simple multilingual lemmatizer for Python, especially useful for speed and efficiency

fundus

A very simple news crawler with a funny name

ms3

A parser for annotated MuseScore 3 files.

align

Python library for extracting quantitative, reproducible metrics of multi-level alignment between speakers in naturalistic language corpora.

opusfilter

Toolbox for filtering parallel corpora

audiomate

Python library for handling audio datasets.

mineru-html

MinerU-HTML: An SLM-powered HTML main content extractor that outputs clean HTML bodies. Perfect for Deep Research Agents, RAG applications, and training data generation.

mfte

MFTE (Multi Feature Tagger of English) Python is the Python version based on Le Foll's MFTE written in Perl. It is extended to include semantic tags from Biber (2006) and Biber et al. (1999), including other specific tags.

corpus-similarity

Measure the similarity of text corpora for 74 languages

concordancer

Searching in-memory corpus with Corpus Query Language (CQL)

lyricscorpora

An unofficial Python API that allows users to create a corpus of lyrical text from their favorite artists and billboard charts

treesearch-ud

High-performance toolkit for querying linguistic dependency parses

ua-gec

UA-GEC: Grammatical Error Correction and Fluency Corpus for the Ukrainian language

betastr

An open source reimplementation of Benny Brodda's BETA in Python