Tiktoken Python Packages

count-tokens

Count tokens in a text file.

pyrwkv-tokenizer

A fast RWKV Tokenizer written in Rust

rs-bpe

A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust

rwkv-tokenizer

A fast RWKV Tokenizer written in Rust

pytokencounter

A simple Python library for tokenizing text and counting tokens. While currently only supporting OpenAI LLMs, it helps with text processing and managing token limits in AI applications.

openai-function-tokens

Predict the exact openai token usage of functions

puretiktoken

tiktoken in pure Python: byte-for-byte compatible BPE tokenizer for GPT-4/4o, no Rust extension, zero dependencies

skim-llm

Runtime token proxy + intelligence dashboard for LLM tools. Intercepts API calls, strips waste, tracks costs — for individual developers and Fortune 500 teams.

chatgpt-long-term-memory

The ChatGPT Long Term Memory package is a powerful tool designed to empower your projects with the ability to handle a large number of simultaneous users and external sources.

runtoken

A blazing-fast BPE tokenizer for LLMs. Drop-in tiktoken replacement, 20-80x faster.

distill-llm

Runtime token proxy + intelligence dashboard for LLM tools. Intercepts API calls, strips waste, tracks costs — for individual developers and Fortune 500 teams.

llm-context-compressor

Shrink LLM context windows 40-80% (input-dependent, see benchmarks) by removing noise, redundancy, and long-tail detail — zero-dependency Python pipeline with tiktoken counting, an extractive summarizer, and a security-scan brief generator.

llm-tokenscope

Profile your LLM payloads. Find the waste. Cut the cost. Field-level token attribution, cost leak detection, and payload optimization for any LLM API.

convopack

Framework-agnostic, provider-agnostic context-window packer for LLM chat history

tokdu

tokdu (Token Disk Usage) is a terminal-based utility that helps you analyze and visualize token usage in your codebase. Similar to the classic du (disk usage) command, tokdu shows you how many tokens your files and directories consume, which is essential when working with Large Language Models (LLMs) that have token limits.

aitc

CLI that counts tokens in text, files or stdin with tiktoken and estimates input cost on GPT, Claude and Gemini models; compares two files and can use live OpenRouter prices.

slash-tokens

Token Optimization for Context Engineers. 4.8 KB WASM. Sub-millisecond. Zero dependencies.

promptminify

Minify LLM prompts to save tokens — domain-aware, tiktoken-validated, zero regressions

token-trim-cli

Trim Python code for LLMs by removing comments, docstrings, and whitespace. Save AI tokens, reduce API costs, and fit more code into ChatGPT, Claude, and Gemini.

tokenmeter-cli

Count tokens and estimate cost for prompts before you send them.

toksight

Tokenizer analysis & comparison toolkit — compression, coverage, audit, cost estimation. Zero deps.

context-budget

Claude Code context window budget calculator — know your token cost before you load

soon-format

SOON Python implementation — soon-format on PyPI