Deduplication Python Packages

recordlinkage

A powerful and modular toolkit for record linkage and duplicate detection in Python

splink

Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends

semhash

Fast Multimodal Semantic Deduplication & Filtering

borgbackup

Deduplicating archiver with compression and authenticated encryption.

nomenklatura

Framework and command-line tools for integrating FollowTheMoney data streams from multiple sources

fastcdc

FastCDC implementation in Python https://pypi.org/project/fastcdc/

goldenmatch

Zero-config entity resolution & record linkage. The zero-tuning Fellegi-Sunter path beats hand-tuned Splink head-to-head and scales from a CSV to a verified 100M-row dedupe in 9.2 min. Fuzzy/exact/probabilistic + PPRL + LLM + identity graph. Python + edge-safe TypeScript (WASM), SQL-native in Postgres & DuckDB, MCP/REST + dbt/Airflow.

goldenmatch-native

Zero-config entity resolution & record linkage. The zero-tuning Fellegi-Sunter path beats hand-tuned Splink head-to-head and scales from a CSV to a verified 100M-row dedupe in 9.2 min. Fuzzy/exact/probabilistic + PPRL + LLM + identity graph. Python + edge-safe TypeScript (WASM), SQL-native in Postgres & DuckDB, MCP/REST + dbt/Airflow.

chantal

A unified CLI tool for offline repository mirroring across multiple package ecosystems (RPM, APT, APK, Helm and more)

removedup

Remove duplicates from parallel corpora

goldenflow

Zero-config entity resolution & record linkage. The zero-tuning Fellegi-Sunter path beats hand-tuned Splink head-to-head and scales from a CSV to a verified 100M-row dedupe in 9.2 min. Fuzzy/exact/probabilistic + PPRL + LLM + identity graph. Python + edge-safe TypeScript (WASM), SQL-native in Postgres & DuckDB, MCP/REST + dbt/Airflow.

pyfastcdc

A high-performance FastCDC 2020 implementation written in Python + Cython

redis-message-queue

Lightweight Python message queuing with Redis and built-in publish-side deduplication.

bankstatementparser

Parse bank statements — CAMT/ISO 20022, PAIN.001, CSV (EN/DE/FR/ES), OFX/QFX, MT940, PDFs (digital + scanned) — into unified Transactions. Deterministic parsers, LLM/vision fallback, Golden Rule + multi-currency + continuity verification, page provenance, dedupe, review, accuracy evals, CLI + REST API. 100% test coverage.

goldencheck

Zero-config entity resolution & record linkage. The zero-tuning Fellegi-Sunter path beats hand-tuned Splink head-to-head and scales from a CSV to a verified 100M-row dedupe in 9.2 min. Fuzzy/exact/probabilistic + PPRL + LLM + identity graph. Python + edge-safe TypeScript (WASM), SQL-native in Postgres & DuckDB, MCP/REST + dbt/Airflow.

blockingpy

Blocking records for record linkage and data deduplication based on ANN algorithms in Python.

redup

Code duplication analyzer and refactoring planner for LLMs

mail-deduplicate

📧 CLI to deduplicate mails from mail boxes

infermap

Zero-config entity resolution & record linkage. The zero-tuning Fellegi-Sunter path beats hand-tuned Splink head-to-head and scales from a CSV to a verified 100M-row dedupe in 9.2 min. Fuzzy/exact/probabilistic + PPRL + LLM + identity graph. Python + edge-safe TypeScript (WASM), SQL-native in Postgres & DuckDB, MCP/REST + dbt/Airflow.

zingg

Scalable master data management, identity resolution, entity resolution, and deduplication using ML

dbretina

DBRetina Python Package

goldenpipe

Zero-config entity resolution & record linkage. The zero-tuning Fellegi-Sunter path beats hand-tuned Splink head-to-head and scales from a CSV to a verified 100M-row dedupe in 9.2 min. Fuzzy/exact/probabilistic + PPRL + LLM + identity graph. Python + edge-safe TypeScript (WASM), SQL-native in Postgres & DuckDB, MCP/REST + dbt/Airflow.

mismo

The SQL/Ibis powered sklearn of record linkage

pyjedai

An open-source library that leverages Python’s data science ecosystem to build powerful end-to-end Entity Resolution workflows.