Pretraining Python Packages

rose-opt

🌹 Rose: Range-Of-Slice Equilibration PyTorch optimizer. Stateless optimization through range-normalized gradient updates.

ptlasso

A Python package for fitting pretrained Lasso models

gpt-simple-lm

A clean, readable framework for pretraining language models from scratch.

forgellm

A comprehensive toolkit for end-to-end continued pre-training, fine-tuning, monitoring, testing and publishing of language models with MLX-LM

decontaminate

`decon`, but with python API binding.

iltm

iLTM: Integrated Large Tabular Model

alea-preprocess

Accessible, efficient data preprocessing library for pretrain and SFT datasets, including KL3M

ccdown

A rust based, resumable downloader cli and python library for Common Crawl data

zeldarose

Train transformer-based models.

proteinworkshop

Benchmarking framework for protein representation learning. Includes a large number of pre-training and downstream task datasets, models and training/task utilities. (ICLR 2024)

autoevolve

Companion tools for Karpathy's autoresearch - smarter evaluation, guided steering, and multi-agent competitions for GPT pretraining

fleet-x

飞桨大模型开发套件，提供大语言模型、跨模态大模型、生物计算大模型等领域的全流程开发工具链。

chinchilla

A toolkit for scaling law research ⚖

autojudge

Companion tools for Karpathy's autoresearch - smarter evaluation, guided steering, and multi-agent competitions for GPT pretraining

autosteer

Companion tools for Karpathy's autoresearch - smarter evaluation, guided steering, and multi-agent competitions for GPT pretraining

graphg

GraphGen: Enhancing Supervised Fine-Tuning for LLMs with Knowledge-Driven Synthetic Data Generation

lumenspark

Lumenspark is a lightweight Linformer-based Language Model Trained from Scratch

fleet-lightning

No description available

ngab

Benchmarking and generating PE for GNNs via the Graph Alignment task. Code for our paper: Graph Alignment for Benchmarking Graph Neural Networks and Learning Positional Encodings

sotastream

A library for data streaming and augmentation

paddle-fleet

飞桨大模型开发套件，提供大语言模型、跨模态大模型、生物计算大模型等领域的全流程开发工具链。