Rag Evaluation Python Packages

giskard

🐢 Open-Source Evaluation & Testing library for LLM Agents

haiku-rag-slim

Agentic RAG for local and self-hosted document search: hybrid retrieval, reranking and multimodal RAG on embedded LanceDB, with Docling parsing and an MCP server

haiku-rag

Agentic RAG for local and self-hosted document search: hybrid retrieval, reranking and multimodal RAG on embedded LanceDB, with Docling parsing and an MCP server

multivon-eval

Practical LLM evaluation for teams that ship to production. Deterministic + LLM-as-judge evaluators, dataset support, CI/CD integration.

proofrag

Point your agent at your docs and your RAG app; get a golden test set + an LLM-as-judge & retrieval scorecard, in one command.

ragscore

⚡️ The "1-Minute RAG Audit" — Generate QA datasets & evaluate RAG systems in Colab, Jupyter, or CLI. Privacy-first, async, visual reports.

contexttrace

Local-first SDK and CLI for debugging RAG failures, verifying citations, classifying failure modes, and generating reliability reports for user-built RAG and AI agent systems.

atheros-compliance-kit

Compliance evidence, generated by the system that needs it — EU AI Act + ISO/IEC 42001 evidence from your own codebase and CI. Python, stdlib-only core, offline.

tunarag-python

TunaRAG is a Python package for tuning LangChain, LangGraph, and Google ADK RAG pipelines to balance answer quality, cost, and latency.

rag-benchmarking

RAG Benchmarking — Framework-agnostic RAG/agentic-AI evaluation harness. Faithfulness, agentic metrics, EU AI Act Article 15 accuracy evidence. Apache 2.0.

kb-arena

Benchmark 19 retrieval architectures (vector, contextual, QnA, knowledge graph, LightRAG, hybrid, RAPTOR, PageIndex, BM25, metadata-filtered, temporal, rerank, HyDE, multi-query, late-interaction, SPLADE, agentic, plus two quantum-inspired rerankers) on your own docs. Citable evidence bundles, bootstrap CIs, significance tests.

llamator

Red Teaming python-framework for testing chatbots and GenAI systems.

openagent-eval

Local-first evaluation framework for RAG systems and AI Agents. 18+ metrics, CLI + SDK, framework-agnostic. The pytest of AI evaluation.

open-rag-eval

RAG evaluation without the need for "golden answers"

hitgate

Portable hybrid (BM25 + dense + RRF) retrieval engine and a label-free evaluation harness — extracted from a personal AI-assistant memory index and decoupled to run on any source tree.

ragrefine

Lightweight retrieval refinement for RAG systems, combining neural reranking, lexical relevance, structured constraint matching, and reproducible evaluation over frozen candidate pools.

mcpaisuite-evalmcp

Evaluation for AI agents — judge-based scoring and native RAG metrics (faithfulness, relevancy, context precision/recall). Python lib · CLI · MCP server · FastAPI.

causa-rag-client

A diagnostic bench for RAG. Scores every question separately, names the pipeline stage that failed, and diffs two runs question by question. Runs entirely on your own machine.

rurage

RURAGE (Robust Universal RAG Evaluation) is a Python library developed to speed-up evaluation of RAG systems with Correctness, Faithfulness and Relevance axes using a variety of deterministic and model-based metrics.

dokis

Lightweight RAG provenance middleware. Verifies every claim in an LLM response is grounded in a retrieved source - without an LLM call.

rag-forge-core

Production-grade RAG pipelines with evaluation baked in

smallevals

smallevals — CPU-fast, GPU-blazing fast offline retrieval evaluation for RAG systems with tiny QA models.

rag-forge-observability

Production-grade RAG pipelines with evaluation baked in

rag-forge-evaluator

Production-grade RAG pipelines with evaluation baked in