Rag Evaluation Python Packages

agenta

The open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM observability all in one place.

giskard

🐢 Open-Source Evaluation & Testing library for LLM Agents

multivon-eval

Practical LLM evaluation for teams that ship to production. Deterministic + LLM-as-judge evaluators, dataset support, CI/CD integration.

autorag

AutoRAG: An Open-Source Framework for Retrieval-Augmented Generation (RAG) Evaluation & Optimization with AutoML-Style Automation

contexttrace

Local-first SDK and CLI for debugging RAG failures, verifying citations, classifying failure modes, and generating reliability reports for user-built RAG and AI agent systems.

kb-arena

Benchmark 9 retrieval architectures (vector, contextual, QnA, knowledge graph, hybrid, RAPTOR, PageIndex, BM25, rerank) on your own docs. Automated hyperparameter search with bootstrap CIs and significance tests.

ragscore

⚡️ The "1-Minute RAG Audit" — Generate QA datasets & evaluate RAG systems in Colab, Jupyter, or CLI. Privacy-first, async, visual reports.

mcpaisuite-evalmcp

Evaluation for AI agents — judge-based scoring and native RAG metrics (faithfulness, relevancy, context precision/recall). Python lib · CLI · MCP server · FastAPI.

proofrag

Point your agent at your docs and your RAG app; get a golden test set + an LLM-as-judge & retrieval scorecard, in one command.

llamator

Framework for testing vulnerabilities of GenAI systems.

veralith

Hallucination diagnosis for RAG systems — Sufficiency, Faithfulness, Completeness verdicts plus rule-based remediation.

open-rag-eval

RAG evaluation without the need for "golden answers"

ragaliq

LLM & RAG evaluation testing framework — hallucination detection, faithfulness metrics, answer relevance scoring, and retrieval pipeline testing with pytest integration

dokis

Trust reporting and provenance enforcement for RAG pipelines. Audits claim grounding, source allowlists, and trust failures without an LLM call.

ragverdict

pytest for RAG agents — behavioral audits with PASS/FAIL/WEAK verdicts

hitgate

Portable hybrid (BM25 + dense + RRF) retrieval engine and a label-free evaluation harness — extracted from a personal AI-assistant memory index and decoupled to run on any source tree.

rag-forge-evaluator

Evaluation engine: RAGAS, DeepEval, LLM-as-Judge, and audit report generation

rag-forge-core

RAG pipeline primitives: ingestion, retrieval, context management, and security

smallevals

smallevals — CPU-fast, GPU-blazing fast offline retrieval evaluation for RAG systems with tiny QA models.

rag-forge-observability

Observability stack: OpenTelemetry tracing, Langfuse integration, and drift detection

rurage

RURAGE (Robust Universal RAG Evaluation) is a Python library developed to speed-up evaluation of RAG systems with Correctness, Faithfulness and Relevance axes using a variety of deterministic and model-based metrics.

vero-eval

The End-to-End LLM Evaluation Framework

rag-benchmarking

Framework-agnostic evaluation harness for RAG and agentic AI systems

longprobe

Sub-second RAG regression testing. Define golden questions, detect lost chunks in CI. pytest for your RAG pipeline.