Multimodal Deep Learning Python Packages

pytorch-widedeep

A flexible package for multimodal-deep-learning to combine tabular data with text and images using Wide and Deep models in Pytorch

scarches

Reference mapping for single-cell genomics

finrobot

FinRobot: An Open-Source AI Agent Platform for Financial Analysis using LLMs 🚀 🚀 🚀

navit-torch

My implementation of "Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution"

mmsa-fet

A Tool for extracting multimodal features from videos.

bitnet

Implementation of "BitNet: Scaling 1-bit Transformers for Large Language Models" in pytorch

swarms-torch

Swarming algorithms like PSO, Ant Colony, Sakana, and more in PyTorch 😊

vitaminp

VitaminP: a vision transformer-assisted multimodal integration network for pathology cell segmentation

pegasusx

PegasusX: The Future of Multimodal Embeddings 🦄 🦄

pali-torch

Democratization of "PaLI: A Jointly-Scaled Multilingual Language-Image Model"

medpalm

MedPalm - Pytorch

loraclip

Easy wrapper for inserting LoRA layers in CLIP.

roboml

Machine learning models optimized for robotics experimentation and deployment

torchmm

PyTorch Data loaders and abstraction for multi-modal data.

multimind-sdk

Your SDK solves all of this. One interface. Unified logic. Local + hosted models. Fine-tuning. Agent tools. Enterprise-ready. Hybrid RAG.Star 🌟 if you like it!

slp

Utils and modules for Speech Language and Multimodal processing using pytorch and pytorch lightning

the-compiler

Seed, Code, Harvest: Grow Your Own App with Tree of Thoughts!

ripple-net

image retrieval/tagging with CLIP

kosmos2-torch

My implementation of Kosmos2.5 from the paper: "KOSMOS-2.5: A Multimodal Literate Model"

cross-attn

MultiModalCrossAttn - Pytorch

pali3

pali3 - Pytorch

mmae

Package for multimodal autoencoders with Bregman divergences.

mmqqa

Experiments around using Multi-Modal Casual Attention with Multi-Grouped Query Attention

vodin

SOTA Classification at scale for UAVs, Drones, and much more