Eval Python Packages

asteval

minimalistic evaluator of python expression using ast module

evalidate

Safe and fast evaluation of untrusted user-supplied python expressions

swival

A small, powerful, open-source CLI coding agent that works with open models.

litereval

Wrapper around ast.literal_eval with additional {foo='bar', key=None} dict syntax. + Deep merge two dictionaries.

polynx

pandas-style query/eval/assign for Polars: string expressions compiled to native Polars expressions. Eager + lazy, @var, UDFs, agent-ready.

ragdesk

Personal, local-first RAG over your own sources — hybrid retrieval, cited answers, published eval numbers.

evalfloor

EvalFloor: is your LLM eval improvement real? Trying k prompt variants and keeping the best scores points on noise alone — this computes how many.

pixie-qa

Agent skill for AI agent development

evalint

Lint an LLM eval set: reliability, wrong-answer items, near-duplicates, and droppable cases.

twinkle-eval

High-performance LLM evaluation framework with parallel API calls — up to 17× faster than sequential tools. Supports box, math, and logit-based evaluation.

redhop

RedHop — a reasoning-aware retrieval & context runtime for RAG (in-process, no vector DB). Light weight alternative to langchain and llamaindex for retrieval alone.

memtrust-cli

Independent CLI benchmark harness for agent-memory backends (MemPalace, Mem0, Zep, OpenViking); publishes raw eval logs.

iris-eval

Stop shipping agents on vibes. Score every agent output for quality, safety, and cost.

slop-eval-cli

CLI (npm + PyPI) and GitHub Action that scores AI-generated UI for genericness using an Anthropic LLM-judge rubric, with a --json mode and library API for CI and agent quality gates

evalport-sdk

EvalPort — The Open LLM Evaluation Standard. Portable test cases, graders, suites, and results across DeepEval, Promptfoo, Inspect AI, and more.

codesafe

An open-source Python library for code encryption, decryption, and safe evaluation using Python's built-in AST module, complete with allowed functions, variables, built-in imports, timeouts, and blocked access to attributes.

fieldtest

LLM evaluation framework — define what correct, well-formed, and safe means before you measure

argon-agents

Argon adapters for AI agent frameworks: LangGraph checkpointer with whole-store fork, Mem0 sandbox factory, reproducible eval pins — versioned MongoDB sandboxes for agents

openjury

An sdk and framework for evaluating and comparing multiple model outputs using configurable LLM-based jurors

trap-cli

tp — score any agent, skill, model or script against an eval task, then submit it to a public leaderboard. Non-invasive: your solution runs as a subprocess, nothing to import. Powers trapstreet.run.

llm-evalgate

Eval gates with error bars: confidence intervals, calibrated LLM judges, and a statistically honest regression gate for LLM pipelines

mcp-curate

Turn an OpenAPI/Swagger spec into a high-quality, curated MCP server — consolidates endpoints into clear tools and proves it with an eval harness.

tooldrift

Snapshot and compare tool-call response contracts across configured LLM endpoints, with offline fixture replay and CI-readable drift reports.

evallint

Audits LLM eval datasets for the flaws that make evaluations silently lie — discrimination failure, near-duplicates, class imbalance.