Llm Evaluation Toolkit Python Packages

calibrate-sdk

Python SDK for Calibrate's API

abslang

Agent Behavior Specification — describe, run, and evaluate AI agent interactions from the command line

parea-ai

Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)

qa-metrics

An easy python package to run quick basic QA evaluations. This package includes standardized QA evaluation metrics and semantic evaluation metrics: Black-box and Open-Source large language model prompting and evaluation, exact match, F1 Score, PEDANT semantic match, transformer match. Our package also supports prompting OPENAI and Anthropic API.

langtest

Deliver safe & effective language models

scalexi

scalexi is a versatile open-source Python library, optimized for Python 3.11+, focuses on facilitating low-code development and fine-tuning of diverse Large Language Models (LLMs).

nlptest

Deliver safe & effective language models

evalsense

Tools for systematic large language model evaluations