Chunking Python Packages

semchunk

A fast, lightweight and easy-to-use Python library for splitting text into semantically meaningful chunks.

fastcdc

FastCDC implementation in Python https://pypi.org/project/fastcdc/

quebra-frases

chunks strings into byte sized pieces

chunkshop

Standalone ingest-to-pgvector: source → chunker → embedder → extractor → table. int8 by default.

betterhtmlchunking

BetterHTMLChunking is a Python library for intelligent HTML segmentation. It builds a DOM tree from raw HTML and extracts content-rich regions of interest, making content analysis effortless. Great for LLM based processing.

chunklet-py

High-fidelity context-aware chunking and interactive visualization for RAG. Advanced segmentation for code and documents, because your LLM is only as smart as the fragments you feed it.

chunkrank

Model-aware text chunking and answer re-ranking for LLM pipelines. Automatically adapts chunk size to tokenizer and context window, then consolidates and ranks answers across chunks.

chunktuner

Benchmark chunking strategies for your RAG corpus. Get a recommended config. CLI, Python library, and MCP server.

longparser

Privacy-first document intelligence engine — parse PDFs, DOCX, PPTX, XLSX & CSV into AI-ready chunks for RAG pipelines. Includes HITL review, 3-layer memory chat, and a production FastAPI server.

table2rules

Convert HTML tables into flat, self-contained facts for LLMs and RAG pipelines.

langformers

🚀 Unified NLP Pipelines for Language Models

contextinator

Filesystem tools for AI agents with optional RAG capabilities

chunklet

A smart multilingual text chunker for LLMs, RAG, and beyond.

faiss-vector-aggregator

This Python library provides a suite of advanced methods for aggregating multiple embeddings associated with a single document or entity into a single representative embedding.

omnichunk

Structure-aware text chunking library for code, prose, and markup files. Intelligently splits files into context-rich chunks while preserving semantic boundaries. Supports 15+ programming languages, deterministic output, and zero external dependencies. Perfect for RAG systems, code analysis, and LLM context optimization.

chonky

Chonky is a Python library that intelligently segments text into meaningful semantic chunks using a fine-tuned transformer model.

llmdocparser

A package for parsing PDFs and analyzing their content using LLMs.

django-query-chunk

Django Query Chunk is used to split big queries into multiple chunks for prevent (too high) memory usage.

seedbraid

Reference-based file reconstruction with CDC chunking, SBD1 binary seed format, and IPFS transport

swarmauri-xmp-gif

GIF handler for embedding and extracting XMP packets in Swarmauri runtimes.

llm-pdf-chunker

LLM-friendly PDF splitter & image optimizer. Chunk PDFs by size and downsample images for RAG/Bedrock.

ragpipe-lite

ragpipe-lite: unified RAG ingestion pipeline (loaders, chunking, embeddings, vector store export).

chunksum

Print FastCDC rolling hash chunks and checksums.

notebooklm-chunker

Heading-aware PDF chunking with resumable source and Studio workflows for turning long documents into interactive NotebookLM learning kits.