Pretraining Python Packages

self-play-pretrain-zero-data

Implementation of Self-Play Pretraining with Zero Data

rose-opt

🌹 Rose: Range-Of-Slice Equilibration PyTorch optimizer. Stateless optimization through range-normalized gradient updates.

ibl-bwb-eval

Benchmarking large-scale pretraining and across-animal transfer in multi-region Neuropixels recordings from the International Brain Laboratory.

iltm

iLTM: Integrated Large Tabular Model

ptlasso

A Python package for fitting pretrained Lasso models

alea-preprocess

Accessible, efficient data preprocessing library for pretrain and SFT datasets, including KL3M

forgellm

A comprehensive toolkit for end-to-end continued pre-training, fine-tuning, monitoring, testing and publishing of language models with MLX-LM

decontaminate

`decon`, but with python API binding.

axiomlm

You import the model's weights and start fine-tuning, unaware that you are missing out on more than 80% of gpu utilization. Axiom gets all the juice out of the box.

tokbin-core

Pretraining data for language models: tokenize a text corpus once into resumable, sha256-verified binary shards, move it to a GPU server, and read random training windows via numpy.memmap. Weighted source mixing, PyTorch DataLoader adapters, CLI. Pretraining only - not for SFT or chat data.

tokbin

Pretraining data for language models: tokenize a text corpus once into resumable, sha256-verified binary shards, move it to a GPU server, and read random training windows via numpy.memmap. Weighted source mixing, PyTorch DataLoader adapters, CLI. Pretraining only - not for SFT or chat data.

graphg

GraphGen: Enhancing Supervised Fine-Tuning for LLMs with Knowledge-Driven Synthetic Data Generation

gpt-simple-lm

A clean, readable framework for pretraining language models from scratch.

chinchilla

A toolkit for scaling law research ⚖

fleet-x

飞桨大模型开发套件，提供大语言模型、跨模态大模型、生物计算大模型等领域的全流程开发工具链。

gleamlm

GleamLM大语言模型全栈训练平台——核心代码用 PyTorch 从零实现，最低仅需 12GB 消费级显卡即可完整训练，另有一套工业代码与手写实现消费同一份数据逐层对照。模型训练支持CLI和可视化训练。

zeldarose

Train transformer-based models.

gefen-x

Gefen-X: A Low Memory Optimized Stochastic Optimizer (Fork)

ccdown

A rust based, resumable downloader cli and python library for Common Crawl data

openlanguagemodel

Readable, composable PyTorch library for language models: build Llama, Qwen, Gemma, DeepSeek, Kimi and 20+ other architectures from plain nn.Modules, then train them on CPU, one GPU, or multi-GPU.

proteinworkshop

Benchmarking framework for protein representation learning. Includes a large number of pre-training and downstream task datasets, models and training/task utilities. (ICLR 2024)

autoevolve

Companion tools for Karpathy's autoresearch - smarter evaluation, guided steering, and multi-agent competitions for GPT pretraining

sqlpup

From-scratch 400M text-to-SQL language model

flatbuild

Build language models from scratch. Simple, modular, and reproducible training for conversational AI.