Duplicate Detection Python Packages

nomic

Nomic Developer API SDK

25K 2K 197

slopo

Embedding-based discovery of duplicated logic for projects too large or messy to maintain reliably

24K 812 29

videohash

Near Duplicate Video Detection (Perceptual Video Hashing) - Get a 64-bit comparable hash-value for any video.

10K 386 65

relate-search

Code similarity search by compression - find near-duplicate files, clone families, and where a pasted snippet came from. No embeddings, no model, no vector database.

3K 1 0

mail-deduplicate

📧 CLI to deduplicate mails from mail boxes

1K 201 46

pyjedai

An open-source library that leverages Python’s data science ecosystem to build powerful end-to-end Entity Resolution workflows.

919 100 13

vs-tiletools

Spatial and Temporal Tiling, Padding, and related Utilities for VapourSynth.

886 1 0

Homogenous Groups (Duplication detection, Frequent Itemsets, etc.)

776 0 0

embeadings

Find duplicate and related issues in Beads and Linear, even worded differently, and the open issues your merged PRs already fixed. Read-only; embeds locally. Listed in the Beads community tools.

748 8 0

nmavail

Async Python CLI tool to check name availability across domains, GitHub, NPM, PyPI, Crates.io, and system packages. Zero-config tool for startups and developers."

455 0 0

duplicate-finder

This Python packages identifies duplicate files in a folder of interest.

357 24 6

dupe-eraser

A command-line tool which automate the deletion of duplicate files based on their hash or perceptual-hash.

264 13 0

simages

Find duplicates and similar images in a folder

248 23 3

deplicate

Advanced Duplicate File Finder for Python

224 82 17

maildir-deduplicate

📧 CLI to deduplicate mails from mail boxes

222 201 46

pydupes

A duplicate file finder like rdfind/fdupes et al that may be faster in environments with millions of files and terabytes of data or over high latency filesystems (e.g. NFS).

214 4 1

clonehunter

CloneHunter finds duplicate code across mixed-language repositories. It runs a semantic retrieval pipeline with model inference and indexing to catch harder, non-trivial duplicate patterns.

207 2 0