Duplicate Detection Python Packages

nomic

Nomic Developer API SDK

42K 2K 196

videohash

Near Duplicate Video Detection (Perceptual Video Hashing) - Get a 64-bit comparable hash-value for any video.

12K 377 62

pyjedai

An open-source library that leverages Python’s data science ecosystem to build powerful end-to-end Entity Resolution workflows.

2K 96 13

slopo

Embedding-based code duplication detector

1K 207 4

fast-suffix-array

O(n) suffix array construction (SA-IS) with LCP arrays, BWT, FM-index, and pattern search. Rust-powered Python bindings.

885 0 0

simages

Find duplicates and similar images in a folder

784 23 3

duplicate-finder

This Python packages identifies duplicate files in a folder of interest.

550 24 5

deplicate

Advanced Duplicate File Finder for Python

489 80 17

issuesort

Triage GitHub issues with AI: detect duplicates by meaning and suggest labels. CLI + GitHub Action. Zero dependencies.

484 3 0

py-image-dedup

CLI utility to find near duplicate images and remove all but the best copy.

436 173 18

clonehunter

CloneHunter finds duplicate code across mixed-language repositories. It runs a semantic retrieval pipeline with model inference and indexing to catch harder, non-trivial duplicate patterns.

363 2 0

pydupes

A duplicate file finder like rdfind/fdupes et al that may be faster in environments with millions of files and terabytes of data or over high latency filesystems (e.g. NFS).

335 4 1

imageduplicatefinder

Software that removes duplicated images from folder2 in respect to folder1

319 0 1

demystify-digipres

Engine for analysis of Siegfried export files and DROID CSV

307 33 5

nmavail

Async Python CLI tool to check name availability across domains, GitHub, NPM, PyPI, Crates.io, and system packages. Zero-config tool for startups and developers."

306 0 0