Language Modeling Python Packages

besser

A Python-based AI-enhanced low-code open-source platform for smart software development

bio-embeddings-tape-proteins

Tasks Assessing Protein Embeddings (TAPE), a set of five biologically relevant semi-supervised learning tasks spread across different domains of protein biology.

tape-proteins

Tasks Assessing Protein Embeddings (TAPE), a set of five biologically relevant semi-supervised learning tasks spread across different domains of protein biology.

gated-state-spaces-pytorch

Implementation of Gated State Spaces, from the paper "Long Range Language Modeling via Gated State Spaces", in Pytorch

generate-sequences

A python package made to generate sequences (greedy and beam-search) from Pytorch (not necessarily HF transformers) models.

surprisal

A unified interface for computing surprisal (log probabilities) from language models! Supports neural, symbolic, and black-box API models.

giganticode-dataprep

A toolkit for pre-processing large source code corpora

lemmes

Lemmatizer for spanish language

emoloss

Python package for the paper: EMO: Earth Mover Distance Optimization for Auto-regressive Language Modeling

codeprep

A toolkit for pre-processing large source code corpora

panini-lang

a linguistic analysis/feature extraction framework, LLM-powered

pyxlstm

PyxLSTM: An efficient and extensible implementation of the xLSTM architecture

simctg

No description available

giganticode-codeprep

A toolkit for pre-processing large source code corpora

rezero

ReZero networks

simctgbeta

No description available