Multimodal Deep Learning Python Packages

pytorch-widedeep

A flexible package for multimodal-deep-learning to combine tabular data with text and images using Wide and Deep models in Pytorch

scarches

Reference mapping for single-cell genomics

finrobot

FinRobot: An Open-Source AI Agent Platform for Financial Analysis using LLMs 🚀 🚀 🚀

mmsa-fet

A Tool for extracting multimodal features from videos.

navit-torch

navit - Pytorch

bitnet

Implementation of "BitNet: Scaling 1-bit Transformers for Large Language Models" in pytorch

swarms-torch

Swarming algorithms like PSO, Ant Colony, Sakana, and more in PyTorch 😊

pali-torch

Democratization of "PaLI: A Jointly-Scaled Multilingual Language-Image Model"

pegasusx

pegasus - Pytorch

loraclip

Easy wrapper for inserting LoRA layers in CLIP.

medpalm

MedPalm - Pytorch

vitaminp

VitaminP: a vision transformer-assisted multimodal integration network for pathology cell segmentation

roboml

RoboML is an aggregator package written for prototyping and deploying open source ML models for robotics

slp

Speech, Language and Multimodal Processing models and utilities in PyTorch

torchmm

PyTorch Data loaders and abstraction for multi-modal data.

the-compiler

Seed, Code, Harvest: Grow Your Own App with Tree of Thoughts!

ripple-net

Text-image search and image tagging library

cross-attn

MultiModalCrossAttn - Pytorch

kosmos2-torch

My implementation of Kosmos2.5 from the paper: "KOSMOS-2.5: A Multimodal Literate Model"

multimind-sdk

Your SDK solves all of this. One interface. Unified logic. Local + hosted models. Fine-tuning. Agent tools. Enterprise-ready. Hybrid RAG.Star 🌟 if you like it!

pali3

pali3 - Pytorch

mmae

Package for Multimodal Autoencoders in TensorFlow / Keras

mmqqa

Experiments around using Multi-Modal Casual Attention with Multi-Grouped Query Attention

mmmgqa

Experiments around using Multi-Modal Casual Attention with Multi-Grouped Query Attention