Dedupe Python Packages

recordlinkage

A powerful and modular toolkit for record linkage and duplicate detection in Python

dedupe

:id: A python library for accurate and scalable fuzzy matching, record deduplication and entity-resolution.

django-super-deduper

Utilities for de-duping Django model instances

mail-deduplicate

📧 CLI to deduplicate mails from mail boxes

zingg

Scalable master data management, identity resolution, entity resolution, and deduplication using ML

maildir-deduplicate

Deduplicate mails from a set of maildir folders.

imgdupes

Identifying and removing near-duplicate images using perceptual hashing.

chkbit

Check your files for data corruption and run quick file deduplication

dedupe-fork-eccovia

:id: A python library for accurate and scalable fuzzy matching, record deduplication and entity-resolution.

superdeduper

A simple interface to datamade/dedupe to make probabilistic record linkage easy.

yadupe

Yet another tool to find and remove duplicate files.

pgdedupe

A simple command line interface to the datamade/dedupe library.

oagdedupe

Developed for Use by NY Office of the Attorney General: A Python library for scalable entity resolution, using active learning to learn blocking configurations, generate comparison pairs, then clasify matches

dedupe-fh

A python library for accurate and scaleable data deduplication and entity-resolution