Deduplication Python Packages

splink

Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends

recordlinkage

A powerful and modular toolkit for record linkage and duplicate detection in Python

semhash

Fast Multimodal Semantic Deduplication & Filtering

borgbackup

Deduplicating archiver with compression and authenticated encryption.

fastcdc

FastCDC implementation in Python https://pypi.org/project/fastcdc/

nomenklatura

Framework and command-line tools for integrating FollowTheMoney data streams from multiple sources

pyfastcdc

A high-performance FastCDC 2020 implementation written in Python + Cython

glyd

Lossless AI compression: 33% less GPU memory, bit for bit.

goldenphonetic

Zero-config entity resolution feeding a durable identity layer: messy records from any source become stable golden entities, a Customer 360 with provenance, merge/split and audit. Fellegi-Sunter beats hand-tuned Splink. Arrow-native/Rust, 250M rows in 11.2 min. Python + edge TypeScript (WASM), SQL-native in Postgres & DuckDB, 97 MCP tools + REST.

goldenfuzz

Zero-config entity resolution feeding a durable identity layer: messy records from any source become stable golden entities, a Customer 360 with provenance, merge/split and audit. Fellegi-Sunter beats hand-tuned Splink. Arrow-native/Rust, 250M rows in 11.2 min. Python + edge TypeScript (WASM), SQL-native in Postgres & DuckDB, 97 MCP tools + REST.

goldenmatch

Zero-config entity resolution feeding a durable identity layer: messy records from any source become stable golden entities, a Customer 360 with provenance, merge/split and audit. Fellegi-Sunter beats hand-tuned Splink. Arrow-native/Rust, 250M rows in 11.2 min. Python + edge TypeScript (WASM), SQL-native in Postgres & DuckDB, 97 MCP tools + REST.

goldenmatch-native

Zero-config entity resolution feeding a durable identity layer: messy records from any source become stable golden entities, a Customer 360 with provenance, merge/split and audit. Fellegi-Sunter beats hand-tuned Splink. Arrow-native/Rust, 250M rows in 11.2 min. Python + edge TypeScript (WASM), SQL-native in Postgres & DuckDB, 97 MCP tools + REST.

removedup

Remove duplicates from parallel corpora

golden-suite

Zero-config entity resolution feeding a durable identity layer: messy records from any source become stable golden entities, a Customer 360 with provenance, merge/split and audit. Fellegi-Sunter beats hand-tuned Splink. Arrow-native/Rust, 250M rows in 11.2 min. Python + edge TypeScript (WASM), SQL-native in Postgres & DuckDB, 97 MCP tools + REST.

redup

Code duplication analyzer and refactoring planner for LLMs

winnow-media

Winnow your media library — organize photos and videos, detect duplicates, and keep the best copies.

relate-search

Code similarity search by compression - find near-duplicate files, clone families, and where a pasted snippet came from. No embeddings, no model, no vector database.

dbretina

DBRetina Python Package

goldencheck

Zero-config entity resolution feeding a durable identity layer: messy records from any source become stable golden entities, a Customer 360 with provenance, merge/split and audit. Fellegi-Sunter beats hand-tuned Splink. Arrow-native/Rust, 250M rows in 11.2 min. Python + edge TypeScript (WASM), SQL-native in Postgres & DuckDB, 97 MCP tools + REST.

redis-message-queue

Lightweight Python message queuing with Redis and built-in publish-side deduplication.

goldenflow-native

Zero-config entity resolution feeding a durable identity layer: messy records from any source become stable golden entities, a Customer 360 with provenance, merge/split and audit. Fellegi-Sunter beats hand-tuned Splink. Arrow-native/Rust, 250M rows in 11.2 min. Python + edge TypeScript (WASM), SQL-native in Postgres & DuckDB, 97 MCP tools + REST.

synology-apm-repo-sdk

Offline, read-only Python SDK, CLI, and TUI for Synology ActiveProtect's backup repository — browse, verify, and export backups

unpacksort

Safely unpack, deduplicate, classify, and sort nested mail and archive content

idempotency-kit

Idempotent request handling for async Python applications