Semantic Deduplication Python Packages

semhash

Fast Multimodal Semantic Deduplication & Filtering

okgv

An agent-native CLI for building balanced, duplicate-free synthetic datasets.

invisible-rabbit

Scalable data pre processing and curation toolkit for LLMs

invisible-unicorn

Scalable Data Preprocessing Tool for Training Large Language Models

lava-ray

Scalable data pre processing and curation toolkit for LLMs