Pre Training Python Packages

helical

A framework for state-of-the-art pre-trained bio foundation models on genomics and transcriptomics modalities.

py-data-juicer

Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷

embedding-condensation

[𝗜𝗖𝗠𝗟 𝟮𝟬𝟮𝟲] Dispersion loss counteracts embedding condensation and improves generalization in small language models

electra-pytorch

A simple and working implementation of Electra, the fastest way to pretrain language models from scratch, in Pytorch

marge-pytorch

Implementation of Marge, Pre-training via Paraphrasing, in Pytorch

mlm-pytorch

An implementation of masked language modeling for Pytorch, made as concise and simple as possible

ps3-torch

Scaling Vision Pre-Training to 4K Resolution

ccdown

A polite and user-friendly downloader for Common Crawl data.

coco-lm-pytorch

Implementation of COCO-LM, Correcting and Contrasting Text Sequences for Language Model Pretraining, in Pytorch