Local Llm Python Packages

mindroom

AI agents that know you and your work, in a chat app anyone can use. Open source, any model, self-host or hosted.

veloxquant-mlx

Fast KV-cache quantization for Apple Silicon (MLX) — 43 research-adapted compression methods with Metal kernels

xllamacpp

xllamacpp - a Python wrapper of llama.cpp

m3-memory

Local-first Memory Framework for AI Agents · 99.2% LongMemEval-S retrieval @ k=10 · Supports Claude · Antigravity · LangChain · Hermes · Gemini · OpenCode · OpenClaw · MCP-native and plugins · Hybrid search (FTS5 + vector + MMR) · GDPR · FIPS 140-3 ready · 100% local (fully offline) or cloud capable

rapid-mlx

Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm) on the same weights.

llama-roofline

Is your llama.cpp decode memory-bandwidth-bound? Find out in one command.

lcode-cli

A coding agent that runs entirely on your own machine: open-weight models, up to 256K context, NVIDIA GPU or Apple Silicon Mac, via Ollama.

mlx-kquant

Native K-quant support for MLX, with a quantization and fine-tuning toolchain for Apple Silicon

soup-cli

Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.

local-deep-research

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

lilbee

The whole local AI stack in one executable: it runs and manages local AI models across every GPU, and it's a search engine you can talk to, with cited answers from your files, code, and the web. MCP server for coding agents, web crawler, TUI, CLI, REST API, Python library. No Ollama or LM Studio needed, works with both.

localharness

An open-source, coding agent harness designed specifically for local LLMs. Designed as everything a user needs to jumpstart their own harness (detailed event steams, logs, and performance data) or quickly replace closed source one (featuring out of the box plugins for web search, image gen, memory, auto research and more in dev).

protocore

Protocore — the agent loop, and nothing else. A protocol-first ReAct runtime for LLM agents in Python 3.12+: context budget, tool surface, compaction, snapshot and resume. No database driver, no HTTP endpoint — everything outward is a Protocol you implement. MPL-2.0.

model-compose

Declarative AI pipelines in one YAML file. Tokens, audio chunks, and video frames flow between isolated components. Compose 100+ components for models, agents, speech, vision, and live broadcast. Run local models, cloud APIs, or both. Inspired by docker-compose.

mtplx

The fastest way to run Qwen 3.8 Flash Next, Qwen 3.8 27B and Ternary Bonsai 2 27B on a Mac: 125 tok/s in OpenCode on an M5 Max, and a 27B model on 16 GB Macs. Native MTP speculative decoding on Apple Silicon, exact at any temperature. OpenAI and Anthropic compatible local server.

jarv

Scriptable, multi-provider AI agent for the terminal — pipe into it, script it, run it on any model, cloud or local

wickra-copilot

Market copilot — an LLM grounded in real order book, liquidation and funding microstructure, not vibes: a local trading assistant that cannot hallucinate, over the Wickra core, in ten languages.

ax-engine

One Mac process. Many models. Real speed. Multi-model LLM serving with prefix reuse, MTP acceleration, and OpenAI APIs — built for Apple Silicon, measured against mlx-lm and llama.cpp.

vllm-mlx

High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.

widemem-ai

Local-first memory layer for LLM apps and agents. Runs offline on Ollama, sentence-transformers and FAISS; OpenAI and Anthropic optional. Importance scoring, contradiction resolution, confidence-aware retrieval.

llama-mdl

Run local llama.cpp servers from a config file, with an optional terminal dashboard

dflash-console

Local Windows control panel for llama-server and DFlash speculative-decoding engines.

quackd

One CLI for all your robots. Connect them, command them, and let them work together, each with an LLM for a brain (Claude, OpenAI, Gemini, Grok, or local via Ollama or vLLM), VLAs for arms and decision LLMs (Jev, Laya, Kev). Drives Microduck, Open Duck Mini, LeRobot, XLeRobot, AlohaMini, ToddlerBot, or any ROS 2 base from a laptop, offboard.

mcp-client-for-ollama

Harness the power of local LLMs with this TUI MCP Client for Ollama. Featuring all core MCP primitives (tools, prompts, resources), agent mode, multi-server, model switching, streaming responses, human-in-the-loop, thinking mode, model params config, system prompts, and saved preferences.