Corpus Tools Python Packages

trafilatura

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

simplemma

Simple multilingual lemmatizer for Python, especially useful for speed and efficiency

fundus

A very simple news crawler with a funny name

ms3

A parser for annotated MuseScore 3 files.

align

Python library for extracting quantitative, reproducible metrics of multi-level alignment between speakers in naturalistic language corpora.

opusfilter

Toolbox for filtering parallel corpora

mineru-html

MinerU-HTML: An SLM-powered HTML main content extractor that outputs clean HTML bodies. Perfect for Deep Research Agents, RAG applications, and training data generation.

concordancer

Extract concordance lines from corpus with CQL

audiomate

Audiomate is a library for working with audio datasets.

mfte

MFTE (Multi Feature Tagger of English) Python is the Python version based on Le Foll's MFTE written in Perl. It is extended to include semantic tags from Biber (2006) and Biber et al. (1999), including other specific tags.

corpus-similarity

Measure the similarity of text corpora for 74 languages

ua-gec

UA-GEC: Grammatical Error Correction and Fluency Corpus for the Ukrainian Language

treesearch-ud

High-performance toolkit for querying linguistic dependency parses

betastr

An open source reimplementation of Benny Brodda's BETA in Python

lyricscorpora

An unofficial Python API that allows users to create a corpus of lyrical text from their favorite artists and billboard charts