Failure Analysis Python Packages

agent-diagnostics

Behavioral taxonomy and annotation framework for coding-agent failures: 40 failure categories across 11 dimensions labeled over 11,995 trials from 4 models on 61 benchmarks, with a queryable dataset backend and pip-installable labeling tool.

evalrx

Autonomous failure investigation, root-cause analysis, and validated interventions for LLMs, VLMs, and agents.

cma-mcp

Executable compound practice loop: capture failures and decisions, surface them at the moment of action, and measure whether warnings prevent repeats. Bash CLI + MCP server.

bugdna

A lightweight library that converts noisy stack traces into deterministic failure signatures.

entropy-loop-core

Failure Compiler for AI agents: turn failed outputs into replayable regression cases.

llm-failure-atlas

A graph-based failure modeling and deterministic detection system for LLM agent runtimes.

criticality-spectrometer

Single-point-of-failure and blast-radius analysis with time-to-substitute curves for AND/OR dependency systems.

evalvitals

Autonomous failure investigation, root-cause analysis, and validated interventions for LLMs, VLMs, and agents.

model-failure-lab

Toolkit for systematically probing, classifying, and debugging failure modes in LLM and RAG systems — reasoning errors, hallucination, and API-level behaviour.