Pre Training Python Packages

helical

A framework for state-of-the-art pre-trained bio foundation models on genomics and transcriptomics modalities.

py-data-juicer

Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷

marge-pytorch

Implementation of Marge, Pre-training via Paraphrasing, in Pytorch

electra-pytorch

A simple and working implementation of Electra, the fastest way to pretrain language models from scratch, in Pytorch

coco-lm-pytorch

COCO - Pytorch

mlm-pytorch

An implementation of masked language modeling for Pytorch, made as concise and simple as possible

ps3-torch

Scaling Vision Pre-Training to 4K Resolution

ccdown

A rust based, resumable downloader cli and python library for Common Crawl data