Llm Evaluation Framework Python Packages

deepeval

The LLM Evaluation Framework

qa-metrics

An easy python package to run quick basic QA evaluations. This package includes standardized QA evaluation metrics and semantic evaluation metrics: Black-box and Open-Source large language model prompting and evaluation, exact match, F1 Score, PEDANT semantic match, transformer match. Our package also supports prompting OPENAI and Anthropic API.

parea-ai

Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)

rhesis-sdk

The testing platform for AI teams. Bring engineers, PMs, and domain experts together to generate tests, simulate (adversarial) conversations, and trace every failure to its root cause.

langfair

LangFair is a Python library for conducting use-case level LLM bias and fairness assessments

rhesis

The testing platform for AI teams. Bring engineers, PMs, and domain experts together to generate tests, simulate (adversarial) conversations, and trace every failure to its root cause.

agentic-security

Agentic LLM Vulnerability Scanner / AI red teaming kit 🧪

fieldtest

LLM evaluation framework — define what correct, well-formed, and safe means before you measure

evalsense

Tools for systematic large language model evaluations

deepevals

The LLM Evaluation Framework

multinear

Develop reliable AI apps

ccheck

A human-friendly framework for testing and evaluating LLMs, RAGs, and chatbots.

llmevals

Eval

vero-eval

The End-to-End LLM Evaluation Framework

mseep-agentic-security

Agentic LLM vulnerability scanner

testllm

The LLM Evaluation Framework

voigt-kampff

Voigt-Kampff — a behavioral safety scoring engine for AI systems. Part of the SAPIEN Framework.

langalf

Agentic LLM vulnerability scanner