Multi Modal Python Packages

modelscope

ModelScope: bring the notion of Model-as-a-Service to life.

agentscope

Build and run agents you can see, understand and trust.

docarray

Represent, send, store and search multimodal data

pyrhubarb

A Python framework for multi-modal document understanding with Amazon Bedrock

medmnist

[pip install medmnist] 18x Standardized Datasets for 2D and 3D Biomedical Image Classification

aicu-scanner

Black-box security scanner for LLM applications — prompt injection, safety bypass, credential leakage

pyvalhalla

Open Source Routing Engine for OpenStreetMap

dalle-pytorch

Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch

cn-clip

Chinese version of CLIP which achieves Chinese cross-modal retrieval and representation generation.

transfusion-pytorch

Pytorch implementation of Transfusion, "Predict the Next Token and Diffuse Images with One Multi-Modal Model", from MetaAI

py-data-juicer

Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷

qwen

My personal implementation of the model from "Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities", they haven't released model code yet sooo...

byaldi

Use late-interaction multi-modal models such as ColPali in just a few lines of code.

pyvalhalla-weekly

High-level bindings to the Valhalla C++ library

switch-transformers

Implementation of Switch Transformers from the paper: "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity"

brainles-preprocessing

preprocessing tools for multi-modal 3D brain imaging

deepke

[EMNLP 2022] An Open Toolkit for Knowledge Graph Extraction and Construction

avitai-artifex

A research-focused modular generative modeling library built on JAX/Flax NNX

vision-llama

Implementation of VisionLLaMA from the paper: "VisionLLaMA: A Unified LLaMA Interface for Vision Tasks" in PyTorch and Zeta

mychatgpt

mychatgpt is a small and useful Python package that provides utils to create OpenAI's GPT conversational agents. This module allows users to have interactive chat with GPT models and keeps track of the chat history. Useful in Python projects as Copilot agent.

mmspao

Multi-modal spatial omics integration with views and combinations.

tiny-gptv

Tiny GPTV - Pytorch

mm-poe

Multiple Choice Reasoning via. Process of Elimination using Multi-Modal Models

move-dl

Multi-omics variational autoencoder