Multimodal Deep Learning Python Packages

pytorch-widedeep

A flexible package for multimodal-deep-learning to combine tabular data with text and images using Wide and Deep models in Pytorch

scarches

Reference mapping for single-cell genomics

mmsa-fet

A Tool for extracting multimodal features from videos.

finrobot

FinRobot: An Open-Source AI Agent Platform for Financial Applications using Large Language Models

bitnet

Implementation of "BitNet: Scaling 1-bit Transformers for Large Language Models" in pytorch

pegasusx

PegasusX: The Future of Multimodal Embeddings 🦄 🦄

roboml

RoboML is an aggregator package written for prototyping and deploying open source ML models for robotics

medpalm

Towards Generalist Biomedical AI

swarms-torch

Swarming algorithms like PSO, Ant Colony, Sakana, and more in PyTorch 😊

multimind-sdk

Your SDK solves all of this. One interface. Unified logic. Local + hosted models. Fine-tuning. Agent tools. Enterprise-ready. Hybrid RAG.Star 🌟 if you like it!

slp

Utils and modules for Speech Language and Multimodal processing using pytorch and pytorch lightning

pali-torch

Democratization of "PaLI: A Jointly-Scaled Multilingual Language-Image Model"

ripple-net

image retrieval/tagging with CLIP

the-compiler

Seed, Code, Harvest: Grow Your Own App with Tree of Thoughts!

navit-torch

My implementation of "Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution"

vitaminp

VitaminP: a vision transformer-assisted multimodal integration network for pathology cell segmentation

pali3

Implementation of PALI3 from the paper PALI-3 VISION LANGUAGE MODELS: SMALLER, FASTER, STRONGER"

cross-attn

The open source implementation of the cross attention mechanism from the paper: "JOINTLY TRAINING LARGE AUTOREGRESSIVE MULTIMODAL MODELS"

mmae

Package for Multimodal Autoencoders in TensorFlow / Keras

mmqqa

Experiments around using Multi-Modal Casual Attention with Multi-Grouped Query Attention

vodin

SOTA Classification at scale for UAVs, Drones, and much more

loraclip

Easy wrapper for inserting LoRA layers in CLIP.

kosmos2-torch

My implementation of Kosmos2.5 from the paper: "KOSMOS-2.5: A Multimodal Literate Model"

torchmm

PyTorch Data loaders and abstraction for multi-modal data.