Tiktoken Python Packages

count-tokens

Count tokens in a text file.

rs-bpe

A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust

pyrwkv-tokenizer

A fast RWKV Tokenizer written in Rust

rwkv-tokenizer

RWKV Tokenizer

pytokencounter

A Python library for tokenizing text and counting tokens using various encoding schemes.

openai-function-tokens

A package to estimate token counts for messages AND functions in openai's chat completion API.

runtoken

A blazing-fast BPE tokenizer for LLMs. Drop-in tiktoken replacement, 20-80x faster.

chatgpt-long-term-memory

The ChatGPT Long Term Memory package is a powerful tool designed to empower your projects with the ability to handle a large number of simultaneous users.

llm-tokenscope

Profile your LLM payloads. Find the waste. Cut the cost. Field-level token attribution, cost leak detection, and payload optimization for any LLM API.

tokdu

tokdu (Token Disk Usage) is a terminal-based utility that helps you analyze and visualize token usage in your codebase. Similar to the classic du (disk usage) command, tokdu shows you how many tokens your files and directories consume, which is essential when working with Large Language Models (LLMs) that have token limits.

promptminify

Minify LLM prompts to save tokens — domain-aware, tiktoken-validated, zero regressions

toksight

Tokenizer analysis & comparison toolkit — compression, coverage, audit, cost estimation. Zero deps.

slash-tokens

Token Optimization for Context Engineers. 4.8 KB WASM. Sub-millisecond. Zero dependencies.

distill-llm

See which files burn your LLM tokens — fix them automatically. CI budget gate + MCP server + adapters for Claude, GPT-4o, Gemini, Ollama.