PyRank

Semantic Deduplication Python Packages

Python packages with the GitHub topic semantic-deduplication. Sorted by monthly downloads, with stars and forks.
MinishLab
semhash

Fast Multimodal Semantic Deduplication & Filtering

122K 974 56
NVIDIA
invisible-rabbit

Scalable data pre processing and curation toolkit for LLMs

151 2K 349
NVIDIA
invisible-unicorn

Scalable data pre processing and curation toolkit for LLMs

72 2K 349
AndreaZoccatelli
okgv

An agent-native CLI for building balanced, duplicate-free synthetic datasets.

58 3 0
NVIDIA
lava-ray

Scalable data pre processing and curation toolkit for LLMs

0 2K 349