Multi Modal Python Packages

modelscope

ModelScope: bring the notion of Model-as-a-Service to life.

agentscope

Build and run agents you can see, understand and trust.

docarray

Represent, send, store and search multimodal data

medmnist

[pip install medmnist] 18x Standardized Datasets for 2D and 3D Biomedical Image Classification

pyrhubarb

A Python framework for multi-modal document understanding with Amazon Bedrock

pyvalhalla

Open Source Routing Engine for OpenStreetMap

cn-clip

Chinese version of CLIP which achieves Chinese cross-modal retrieval and representation generation.

py-data-juicer

Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷

dalle-pytorch

Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch

byaldi

Use late-interaction multi-modal models such as ColPali in just a few lines of code.

transfusion-pytorch

Pytorch implementation of Transfusion, "Predict the Next Token and Diffuse Images with One Multi-Modal Model", from MetaAI

qwen

My personal implementation of the model from "Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities", they haven't released model code yet sooo...

avitai-artifex

A research-focused modular generative modeling library built on JAX/Flax NNX

pyvalhalla-weekly

Open Source Routing Engine for OpenStreetMap

deepke

DeepKE is a knowledge extraction toolkit for knowledge graph construction supporting low-resource, document-level and multimodal scenarios for entity, relation and attribute extraction.

brainles-preprocessing

preprocessing tools for multi-modal 3D brain imaging

vision-llama

Implementation of VisionLLaMA from the paper: "VisionLLaMA: A Unified LLaMA Interface for Vision Tasks" in PyTorch and Zeta

switch-transformers

Implementation of Switch Transformers from the paper: "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity"

mychatgpt

mychatgpt is a small and useful Python package that provides utils to create OpenAI's GPT conversational agents. This module allows users to have interactive chat with GPT models and keeps track of the chat history. Useful in Python projects as Copilot agent.

move-dl

Multi-omics variational autoencoder

tiny-gptv

Tiny GPTV - Pytorch

dalle-pytorch-dev

DALL-E - Pytorch

mm1-torch

PyTorch Implementation of the paper "MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training"

simba-torch

A simpler Pytorch + Zeta Implementation of the paper: "SiMBA: Simplified Mamba-based Architecture for Vision and Multivariate Time series"