Multimodal Learning Python Packages

autobotlib

An autoML for explainable text classification.

ak-ssl

A python library for self-supervised learning

immuno-compass

Generalizable AI predicts immunotherapy outcomes across cancers and treatments

biosnn-bus

脉冲神经网络认知原型：纯局部学习规则（无代理梯度）+ SNN 自主调度 LLM。Spiking neural network cognitive prototype — purely local learning rules (no surrogate gradients), SNN autonomously schedules an LLM, modality-plugin architecture. 研究原型，非生产框架 / research prototype, not a production framework.

isynctab

iSyncTab learns synchronized cross-modal feature sequences for image-tabular and audio-video multimodal learning using neural synchrony, NS-PFS, and an order-aware Transformer.

topicnet

Interface for easier topic modelling.

multimodal-transformers

Multimodal model for text and tabular data with HuggingFace transformers as building block for text data

pykale

Knowledge-Aware machine LEarning (KALE): accessible machine learning from multiple sources for interdisciplinary research, part of the 🔥PyTorch ecosystem. ⭐ Star to support our work!

slp

Utils and modules for Speech Language and Multimodal processing using pytorch and pytorch lightning

autort-swarms

Implementation of AutoRT: "AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents"

navit-torch

My implementation of "Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution"

pali3

Implementation of PALI3 from the paper PALI-3 VISION LANGUAGE MODELS: SMALLER, FASTER, STRONGER"

proto-clip-toolkit

Code release for Proto-CLIP: Vision-Language Prototypical Network for Few-Shot Learning | IROS 2024

mmae

Package for Multimodal Autoencoders in TensorFlow / Keras

istructtab

iStructTab: Structured Feature Sequencing for Multimodal Image-Tabular Learning with GEDS and OEMT.

kosmosg

My implementation of the model KosmosG from "KOSMOS-G: Generating Images in Context with Multimodal Large Language Models"

coin-clip

Coin-CLIP: fine-tuned with a vast collection of coin images from CLIP using contrastive learning. It enhances feature extraction for coins, boosting image search accuracy. This model merges Visual Transformer (ViT) with CLIP's multimodal learning, optimized for numismatic applications.