Eval Python Packages

asteval

minimalistic evaluator of python expression using ast module

evalidate

Safe and fast evaluation of untrusted user-supplied python expressions

swival

A small, powerful, open-source CLI coding agent that works with open models.

redhop

RedHop — a reasoning-aware retrieval & context runtime for RAG (in-process, no vector DB). Light weight alternative to langchain and llamaindex for retrieval alone.

litereval

Wrapper around ast.literal_eval with additional {foo='bar', key=None} dict syntax. + Deep merge two dictionaries.

mcp-curate

Turn an OpenAPI/Swagger spec into a high-quality, curated MCP server — consolidates endpoints into clear tools and proves it with an eval harness.

llm-evalgate

Eval gates with error bars: confidence intervals, calibrated LLM judges, and a statistically honest regression gate for LLM pipelines

pixie-qa

Agent skill for AI agent development

ai-crucible

A diagnostic adversarial game for frontier LLMs — a measurement instrument that happens to be fun.

siggate

The vendor-neutral significance gate for AI evals. Most eval tools show a number moving; siggate tells you — in CI — whether the move is statistically real or within noise.

oddish

Run Harbor tasks in the cloud

proofagent

pytest for AI agents — test safety, accuracy, tool use, and cost. No YAML, no telemetry, just Python.

agenttrace

AgentTrace is a lightweight observability library to trace and evaluate agentic systems.

fieldtest

LLM evaluation framework — define what correct, well-formed, and safe means before you measure

model-failure-lab

Toolkit for systematically probing, classifying, and debugging failure modes in LLM and RAG systems — reasoning errors, hallucination, and API-level behaviour.

cval

A layer of protection for python's eval

shipready

Rubric based ship-readiness evals for AI agents.

twinkle-eval

High-performance LLM evaluation framework with parallel API calls — up to 17× faster than sequential tools. Supports box, math, and logit-based evaluation.

codesafe

An open-source Python library for code encryption, decryption, and safe evaluation using Python's built-in AST module, complete with allowed functions, variables, built-in imports, timeouts, and blocked access to attributes.

contextops-tool

Cache-aware prompt structure optimizer + LLM-as-judge eval + local cost/usage logger

sphinxcontrib-eval

Evaluate shell command or python code in sphinx and myst

llmesh-lleval

Reserved name — lleval: LLM eval framework for the FullSense LLM family (on-prem + cloud unified A/B, progressive size matrix, honest disclosure). Reference implementation in development.

aigc-evals

aigc_evals

promptmin

Prompt minimizer for LLM evals — shrink prompts to minimal failing inputs using delta debugging