Dedup Python Packages

ilytix

A CLI tool for images analysis: checking image integrity, images deduplication, image retrieval.

chunkdup

Find (partial content) duplicate files.

py-image-dedup

A library to find duplicate images and delete unwanted ones

merlin-langchain

Merlin Community Edition — free dedup engine + integrations. Saves LLM tokens. No telemetry.

chunksum

Print FastCDC rolling hash chunks and checksums.

chkbit

Check your files for data corruption and run quick file deduplication

yadupe

Yet another tool to find and remove duplicate files.