Dedup Python Packages

ilytix

A CLI tool for images analysis: checking image integrity, images deduplication, image retrieval.

py-image-dedup

CLI utility to find near duplicate images and remove all but the best copy.

chunkdup

Find (partial content) duplicate files.

chunksum

Print FastCDC rolling hash chunks and checksums.

chkbit

Check your files for data corruption and run quick file deduplication

merlin-langchain

Merlin dedup integration for LangChain - strip byte-redundant context before it reaches the LLM.

yadupe

Yet another tool to find and remove duplicate files.