Dedupe Python Packages

recordlinkage

A powerful and modular toolkit for record linkage and duplicate detection in Python

dedupe

:id: A python library for accurate and scalable fuzzy matching, record deduplication and entity-resolution.

find-dup-defs

Find the copy-pasted code your linter can't — rank it by how worth refactoring it is — and surface the recurring shapes that should become one helper (Python/TS/Rust)

django-super-deduper

Utilities for de-duping Django model instances

zingg

Scalable master data management, identity resolution, entity resolution, and deduplication using ML

mail-deduplicate

📧 CLI to deduplicate mails from mail boxes

imgdupes

Identifying and removing near-duplicate images using perceptual hashing.

chkbit

Check your files for data corruption and run quick file deduplication

maildir-deduplicate

📧 CLI to deduplicate mails from mail boxes

dedupe-fork-eccovia

:id: A python library for accurate and scalable fuzzy matching, record deduplication and entity-resolution.

superdeduper

A simple command line interface to the datamade/dedupe library.

oagdedupe

Developed for Use by NY Office of the Attorney General: A Python library for scalable entity resolution, using active learning to learn blocking configurations, generate comparison pairs, then clasify matches

yadupe

Yet another tool to find and remove duplicate files.

pgdedupe

A simple command line interface to the datamade/dedupe library.

dedupe-fh

:id: A python library for accurate and scalable fuzzy matching, record deduplication and entity-resolution.