Pretraining Python Packages

rose-opt

🌹 Rose: Range-Of-Slice Equilibration PyTorch optimizer. Stateless optimization through range-normalized gradient updates.

alea-preprocess

Accessible, efficient data preprocessing library for pretrain and SFT datasets, including KL3M

forgellm

A comprehensive toolkit for end-to-end continued pre-training, fine-tuning, monitoring, testing and publishing of language models with MLX-LM

decontaminate

`decon`, but with python API binding.

zeldarose

Train transformer-based models.

ccdown

A rust based, resumable downloader cli and python library for Common Crawl data

iltm

iLTM: Integrated Large Tabular Model

fleet-x

No description available

chinchilla

A toolkit for scaling law research ⚖

autoevolve

Multi-agent research competition orchestrator for autoresearch

proteinworkshop

Benchmarking framework for protein representation learning. Includes a large number of pre-training and downstream task datasets, models and training/task utilities. (ICLR 2024)

autojudge

Smarter experiment evaluation for autoresearch — replaces eyeballing val_bpb with statistical verdicts

autosteer

Companion tools for Karpathy's autoresearch - smarter evaluation, guided steering, and multi-agent competitions for GPT pretraining

graphg

GraphGen: Enhancing Supervised Fine-Tuning for LLMs with Knowledge-Driven Synthetic Data Generation

fleet-lightning

飞桨大模型开发套件，提供大语言模型、跨模态大模型、生物计算大模型等领域的全流程开发工具链。

lumenspark

Lumenspark is a lightweight Linformer-based Language Model Trained from Scratch

ngab

Benchmarking and generating PE for GNNs via the Graph Alignment task. Code for our paper: Graph Alignment for Benchmarking Graph Neural Networks and Learning Positional Encodings

sotastream

A library for data streaming and augmentation

paddle-fleet

飞桨大模型开发套件，提供大语言模型、跨模态大模型、生物计算大模型等领域的全流程开发工具链。