Duplicate Detection Python Packages

nomic

Nomic Developer API SDK

videohash

Near Duplicate Video Detection (Perceptual Video Hashing) - Get a 64-bit comparable hash-value for any video.

pyjedai

An open-source library that leverages Python’s data science ecosystem to build powerful end-to-end Entity Resolution workflows.

simages

Find similar images in a dataset

duplicate-finder

This Python packages identifies duplicate files in a folder of interest.

nmavail

Async Python CLI tool to check name availability across domains, GitHub, NPM, PyPI, Crates.io, and system packages. Zero-config tool for startups and developers."

deplicate

Advanced Duplicate File Finder for Python. Nothing is impossible to solve.

py-image-dedup

A library to find duplicate images and delete unwanted ones

clonehunter

Find duplicate code in mixed-language repositories using semantic and lexical similarity

imageduplicatefinder

Simple duplication finder for Images, matches on names and then compares image hashes.

pydupes

A duplicate file finder that may be faster in environments with millions of files and terabytes of data.

demystify-digipres

engine for the analysis of DROID and Siegfried file format reports

thebear

Bear - the decluttering deduplicator

imgdups

Very fast image duplicate finder with pickle and cv2

findsame

Find duplicate files and directories using hashes and a Merkle tree

dhunter

Fast, content based duplicate file detector with cache and more!

fast-suffix-array

O(n) suffix array construction (SA-IS) with LCP arrays, BWT, FM-index, and pattern search. Rust-powered Python bindings.

deplicate-cli

Command Line Interface for deplicate.

dupeutil

A command-line program written in Python for detecting and removing duplicate files.

dupe-eraser

A command-line tool which automate the deletion of duplicate files based on their hash or perceptual-hash.

doc2term

A fast sentence/word tokenizer, and punctuation remover.

searchdups

Search for duplicate files