Model Inference Python Packages

openllm

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

mblt-vision-python

Python toolkit for preprocessing, inference, and postprocessing of computer vision models on Mobilint NPUs and ONNX Runtime.

openllm-client

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

openllm-core

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

transformers-mblt

Python toolkit for running pre-quantized Hugging Face Transformers models (LLMs, VLMs, speech recognition) on Mobilint NPUs.

tensortorrent

PyTorch compiler and Rust runtime for multi-CPU/GPU inference, NUMA-aware scheduling, model serving, parameter streaming, and activation spill.

melotts-mblt

Python toolkit for MeloTTS text-to-speech (English and Korean) on Mobilint NPUs, with a CLI and Gradio WebUI.

clip-api-service

CLIP as a service - Embed image and sentences, object recognition, visual reasoning, image classification and reverse image search

stagecoachml

Build two-stage models when your features arrive in two batches at different times.

flexpredict

Schema-aware inference composition for Python with unified outputs, preprocessing, model loading, and heterogeneous ensemble support.

lightinfer

Make your model inference to http server. 一个为“同步重计算模型”量身设计的异步推理服务桥接器。