Vision Language Model Python Packages

mlx-vlm

MLX-VLM is a package for inference and fine-tuning of Vision Language Models (VLMs) on your Mac using MLX.

colpali-engine

The code used to train and run inference with the ColVision models, e.g. ColPali, ColQwen2, and ColSmol.

lmms-eval

One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks

mlx-tune

Fine-tune LLMs on your Mac with Apple Silicon. SFT, DPO, GRPO, Vision, TTS, STT, Embedding, and OCR fine-tuning — natively on MLX. Unsloth-compatible API.

aicu-scanner

Black-box security scanner for LLM applications — prompt injection, safety bypass, credential leakage

x-anylabeling-cvhub

Effortless data labeling with AI support from Segment Anything and other awesome models.

vidore-benchmark

Vision Document Retrieval (ViDoRe): Benchmark. Evaluation code for the ColPali paper.

pdfhell

PDF Hell — adversarial PDFs that break AI document readers. Procedural ground truth, not LLM-as-judge.

capevalkit

Reproducible, all-in-one image captioning evaluation for VLMs.

thepipe-api

Get clean data from tricky documents, powered by vision-language models ⚡

mbodied

Seamlessly integrate state-of-the-art transformer models into robotics stacks

tinydoc

Python SDK for TinyDoc-VLM document understanding — the world's smallest document-specialist VLM

urban-worm

Workflow of reproducible multimodal inference for urban environment evaluation.

nerif

LLM powered Python

kanoa

AI-powered interpretation of data science outputs with multi-backend support (Molmo, Gemini, Claude, OpenAI)

kani-vision

Kani extension for supporting vision-language models (VLMs). Comes with model-agnostic support for GPT-Vision and LLaVA.

llava-torch

[NeurIPS'23 Oral] Visual Instruction Tuning (LLaVA) built towards GPT-4V level capabilities and beyond.

gptparse

A tool for converting PDF documents to Markdown using OCR and vision language models

kani-multimodal-core

Core shared libraries for multimodal Kani extensions.

visionzip

Official repository for VisionZip (CVPR 2025)

ps3-torch

Scaling Vision Pre-Training to 4K Resolution

gaze-vlm

Grounded agentic framework for medical vision-language models: viewer-level image tools, multi-turn tool use, and PubMed/Open-i literature retrieval

german-ocr

German-OCR is specifically trained to extract text from German documents including invoices, receipts, forms, and other business documents.

openscenesense-ollama

OpenSceneSense Ollama is a Python library that harnesses AI for advanced local video analysis, offering customizable frame and audio insights for dynamic applications in media, education, and content moderation.