Ai Reliability Python Packages

openvals

Open-source AI model evaluation and benchmarking framework for LLMs (OpenAI, Ollama, Claude, Gemini)

lintlang

lintlang is a static linter for AI agent configs, tool descriptions, and system prompts that runs zero-LLM quality gating in CI. Catches language-level failures (vague tool descriptions, missing stop conditions, schema gaps) before they reach runtime, with deterministic regex + structural detectors and no model calls.

py-toolguard

The "Cloudflare for AI Agents". 7-layer security interceptor, real-time observability dashboard, and automated reliability testing for MCP and AI tool chains. Prevent hallucinations, prompt injection, and destructive tool calls.

replayd

Turn failed AI agent runs into replayable regression tests

llm-sentry

Unified AI reliability platform. One install, 12 diagnostic engines. Continuous monitoring, fault diagnosis, and compliance for LLM pipelines.

agent-convergence-scorer

Score how similar N agent outputs are — exact match, Jaccard token overlap, divergence point, composite 0-1 score. Stdlib-only.

cogito-ergo

fidelis is zero-LLM agent memory for Claude Code and AI agents: a local-first memory layer whose default retrieval path uses BM25, dense vectors, and reciprocal rank fusion with no LLM call. It returns your original passages verbatim instead of paraphrasing and runs fully local. Benchmarked on LongMemEval-S. MIT, by Hermes Labs.

workflowbench

Lightweight benchmark harness for AI-driven business workflows

pygate-ci

Python quality gate CLI for Ruff, Pyright, and pytest with bounded auto-repair and escalation artifacts

chatbot-auditor

Detect the 7 ways AI chatbots silently fail customers

zer0lint

zer0lint is a memory-extraction health diagnostic for mem0 configs and HTTP memory endpoints. It flags silent failure modes where ingestion reports success but facts never survive the LLM extraction step, then generates a stronger extraction prompt validated on your own model. Works over HTTP with any add/search memory API.

loopgrid

Control Plane for AI Decision Reliability

langquant

LPCI: Statefulness Through Language for Stateless Models

rule-audit

Detect logical contradictions, gaps, and exploitable edge cases in AI system prompts

hermes-rubric

Evidence-first structured scoring. Class-aware rubric templates for deterministic dim sets across runs.

zer0dex

zer0dex is a local dual-layer memory pattern for AI agents: a compressed, human-readable markdown index plus a vector store queried automatically before each message. Built for cross-project recall and cross-reference where flat memory files or vector-only RAG fall short. Local-first, low-latency. Reference implementation by Hermes Labs.