Tiktoken Python Packages

count-tokens

Count tokens in a text file.

rs-bpe

A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust

pyrwkv-tokenizer

A fast RWKV Tokenizer written in Rust

rwkv-tokenizer

RWKV Tokenizer

pytokencounter

A Python library for tokenizing text and counting tokens using various encoding schemes.

puretiktoken

tiktoken in pure Python: byte-for-byte compatible BPE tokenizer for GPT-4/4o, no Rust extension, zero dependencies

skim-llm

Runtime token proxy + optimization toolkit for LLM developers and enterprises. Intercepts API calls, strips waste in real-time, tracks costs, and serves a web dashboard.

runtoken

A blazing-fast BPE tokenizer for LLMs. Drop-in tiktoken replacement, 20-80x faster.

context-budget

Claude Code context window budget calculator — know your token cost before you load

convopack

Framework-agnostic, provider-agnostic context-window packer for LLM chat history.

openai-function-tokens

A package to estimate token counts for messages AND functions in openai's chat completion API.

llm-context-compressor

Shrink LLM context windows by removing noise, redundancy, and long-tail detail — without losing signal.

distill-llm

Runtime token proxy + intelligence dashboard for LLM tools. Intercepts API calls, strips waste, tracks costs — for individual developers and Fortune 500 teams.

chatgpt-long-term-memory

The ChatGPT Long Term Memory package is a powerful tool designed to empower your projects with the ability to handle a large number of simultaneous users.

llm-tokenscope

Profile your LLM payloads. Find the waste. Cut the cost.

tokdu

tokdu (Token Disk Usage) is a terminal-based utility that helps you analyze and visualize token usage in your codebase. Similar to the classic du (disk usage) command, tokdu shows you how many tokens your files and directories consume, which is essential when working with Large Language Models (LLMs) that have token limits.

slash-tokens

Token budget estimation in 4.8 KB WASM. Sub-millisecond. Zero dependencies.

promptminify

Minify LLM prompts to save tokens — domain-aware, tiktoken-validated, zero regressions.

toksight

Tokenizer analysis & comparison toolkit — compression, coverage, audit, cost estimation. Zero deps.