Pre Training Python Packages

helical

A framework for state-of-the-art pre-trained bio foundation models on genomics and transcriptomics modalities.

py-data-juicer

Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷

dantinox

DantinoX: A modular, memory-efficient Transformer implementation in JAX/Flax NNX. Includes Sparse MoE, GQA, Sliding Window Attention, Gradient Accumulation and Checkpointing

ps3-torch

Scaling Vision Pre-Training to 4K Resolution

marge-pytorch

Implementation of Marge, Pre-training via Paraphrasing, in Pytorch

ccdown

A rust based, resumable downloader cli and python library for Common Crawl data

embedding-condensation

[𝗜𝗖𝗠𝗟 𝟮𝟬𝟮𝟲] Dispersion loss counteracts embedding condensation and improves generalization in small language models

mlm-pytorch

An implementation of masked language modeling for Pytorch, made as concise and simple as possible

electra-pytorch

A simple and working implementation of Electra, the fastest way to pretrain language models from scratch, in Pytorch

coco-lm-pytorch

Implementation of COCO-LM, Correcting and Contrasting Text Sequences for Language Model Pretraining, in Pytorch

ps3-python

Scaling Vision Pre-Training to 4K Resolution