Minhash Python Packages

datasketch

MinHash, LSH, LSH Forest, Weighted MinHash, HyperLogLog, HyperLogLog++, LSH Ensemble and HNSW

rensa

High-performance MinHash implementation in Rust with Python bindings for efficient similarity estimation and deduplication of large datasets

sourmash

Quickly search, compare, and analyze genomic and metagenomic data sets.

pychase

PyChase - Python duplicate code detector using AST normalization + MinHash LSH. Catches Type-1/2/3 clones in functions, methods, and classes with zero dependencies.

relate-search

Code similarity search by compression - find near-duplicate files, clone families, and where a pasted snippet came from. No embeddings, no model, no vector database.

llm-sketchkit

Count distinct LLM users and find top token consumers across services, days and tools. Fast, small, portable summaries in Go and Python.

genome-comparator

Fast genome comparison with Mash, from assemblies or reads: distance matrix, UPGMA/NJ/ME trees with bootstrap support, and interactive PCoA

promptlsh

A portable similarity digest for prompt attacks, so reworded jailbreaks correlate instead of reading as unrelated items.

gaoya

Locality Sensitive Hashing

libmhcuda

Weighted MinHash implementation on CUDA (multi-gpu).

pari-similarity

Similarity indexing and deduplication for large datasets, with Rust, Python, and CLI interfaces.

sqsketch

Fixed-size sketches of probability distributions: Bhattacharyya coefficient and Hellinger distance from ~1 KB per profile, with error independent of alphabet size. Library + fully verified paper (every number traced to a committed script).

akin

Python library for detecting near duplicate texts in a corpus at scale.

websieve

Turn a web crawl into an ML-ready dataset. Boilerplate extraction, Gopher/C4 quality filtering, MinHash near-dedup, sharded output. Zero dependencies.

mashing-pumpkins

Minhash and maxhash library in Python, combining flexibility, expressivity, and performance.

leadmatch

Fast lead & contact deduplication for Python. Blocking + MinHash LSH + RapidFuzz: ~148× faster than all-pairs fuzzy matching, same accuracy, fully reproducible benchmark

xiangsi

中文文本相似度计算器

sketch-ds

C++ Implementations of sketch data structures with SIMD Parallelism, including Python bindings