Dedupe Python Packages

recordlinkage

A powerful and modular toolkit for record linkage and duplicate detection in Python

dedupe

:id: A python library for accurate and scalable fuzzy matching, record deduplication and entity-resolution.

zingg

Scalable identity resolution, entity resolution, data mastering and deduplication using ML

django-super-deduper

Utilities for de-duping Django model instances

mail-deduplicate

📧 CLI to deduplicate mails from mail boxes

imgdupes

Identifying and removing near-duplicate images using perceptual hashing.

maildir-deduplicate

📧 CLI to deduplicate mails from mail boxes

dedupe-fork-eccovia

:id: A python library for accurate and scalable fuzzy matching, record deduplication and entity-resolution.

chkbit

Check your files for data corruption and run quick file deduplication

superdeduper

A simple command line interface to the datamade/dedupe library.

yadupe

Yet another tool to find and remove duplicate files.

oagdedupe

Developed for Use by NY Office of the Attorney General: A Python library for scalable entity resolution, using active learning to learn blocking configurations, generate comparison pairs, then clasify matches

pgdedupe

A simple command line interface to the datamade/dedupe library.

dedupe-fh

:id: A python library for accurate and scalable fuzzy matching, record deduplication and entity-resolution.