Llm Evaluation Framework Python Packages

deepeval

The LLM Evaluation Framework

calibrate-sdk

Python SDK for Calibrate's API

abslang

Agent Behavior Specification — describe, run, and evaluate AI agent interactions from the command line

parea-ai

Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)

qa-metrics

An easy python package to run quick basic QA evaluations. This package includes standardized QA evaluation metrics and semantic evaluation metrics: Black-box and Open-Source large language model prompting and evaluation, exact match, F1 Score, PEDANT semantic match, transformer match. Our package also supports prompting OPENAI and Anthropic API.

langfair

LangFair is a Python library for conducting use-case level LLM bias and fairness assessments

agentic-security

Agentic LLM Vulnerability Scanner / AI red teaming kit 🧪

fieldtest

LLM evaluation framework — define what correct, well-formed, and safe means before you measure

ccheck

MIT-licensed Framework for LLMs, RAGs, Chatbots testing. Configurable via YAML and integrable into CI pipelines for automated testing.

evalsense

Tools for systematic large language model evaluations

llmevals

The LLM Evaluation Framework

multinear

Develop reliable AI apps

testllm

The LLM Evaluation Framework

vero-eval

Open source framework for evaluating AI Agents

mseep-agentic-security

Agentic LLM Vulnerability Scanner / AI red teaming kit 🧪

deepevals

The LLM Evaluation Framework

langalf

Agentic LLM Vulnerability Scanner / AI red teaming kit 🧪