Llm Evaluation Framework Python Packages

deepeval

The LLM Evaluation Framework

qa-metrics

An easy python package to run quick basic QA evaluations. This package includes standardized QA evaluation metrics and semantic evaluation metrics: Black-box and Open-Source large language model prompting and evaluation, exact match, F1 Score, PEDANT semantic match, transformer match. Our package also supports prompting OPENAI and Anthropic API.

parea-ai

Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)

rhesis-sdk

The testing platform for AI teams. Bring engineers, PMs, and domain experts together to generate tests, simulate (adversarial) conversations, and trace every failure to its root cause.

agentic-security

Agentic LLM Vulnerability Scanner / AI red teaming kit 🧪

langfair

LangFair is a Python library for conducting use-case level LLM bias and fairness assessments

rhesis

The testing platform for AI teams. Bring engineers, PMs, and domain experts together to generate tests, simulate (adversarial) conversations, and trace every failure to its root cause.

fieldtest

LLM evaluation framework — define what correct, well-formed, and safe means before you measure

evalsense

Tools for systematic large language model evaluations

multinear

Develop reliable AI apps

ccheck

A human-friendly framework for testing and evaluating LLMs, RAGs, and chatbots.

llmevals

The LLM Evaluation Framework

deepevals

The LLM Evaluation Framework

vero-eval

Open source framework for evaluating AI Agents

mseep-agentic-security

Agentic LLM vulnerability scanner

testllm

The LLM Evaluation Framework

langalf

Agentic LLM Vulnerability Scanner / AI red teaming kit 🧪