Llm As Judge Python Packages

eval-harness

A boring, config-driven harness for evaluating AI systems. One YAML drives the run, the trace is the source of truth. Offline, backtesting, and online-eval modes — works with any agent, RAG, or code-modifying system.

autosynth

Agentic synthetic-data generation framework inspired by Meta FAIR's Autodata / Agentic Self-Instruct.

judge-bench

Bias probes and reproducible diagnostics for LLM-as-judge evaluation workflows.

judge-card

A disclosure format for judge prompts, calibration results, known bias, and recommended use envelopes.

pyxtaf

Agentic Extensible Test Automation Framework

cortex-research-suite

[Legacy] AI Research OS — 27 self-evolving skills. Decomposed into 20 repos at github.com/TECHKNOWMAD-LABS

ragaliq

LLM & RAG evaluation testing framework — hallucination detection, faithfulness metrics, answer relevance scoring, and retrieval pipeline testing with pytest integration