PyRank
  • Search
  • PyPI
  • GitHub
  • Insights
  • Compare
  • Advisories
  • Ecosystem
  • About

Text Tokenization Python Packages

Python packages with the GitHub topic text-tokenization. Sorted by monthly downloads, with stars and forks.
twardoch
split-markdown4gpt

A Python tool for splitting large Markdown files into smaller sections based on a specified token limit. This is particularly useful for processing large Markdown files with GPT models, as it allows the models to handle the data in manageable chunks.

11K 29 3
alasdairforsythe
tokenmonster

Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript

1K 626 23
    • Original data from PyPI, GitHub, and OSV; downloads served through ClickHouse
    • Sources & methodology