Experimentation Python Packages

optimizely-sdk

Python SDK for Optimizely Feature Experimentation and Optimizely Full Stack (legacy).

gofeatureflag-python-provider

GO Feature Flag is a simple, complete and lightweight self-hosted cloud native feature flag solution 100% Open Source — built on OpenFeature 🎛️

rapidfireai

RapidFire AI: Rapid AI Customization from RAG to Fine-Tuning

experiment-utils-pd

Generic functions for experiment analysis and design

moptipy

Implementation of metaheuristic optimization methods in Python for scientific, industrial, and educational scenarios. Experiments can be executed in parallel or in a distributed fashion. Experimental results can be evaluated in various ways, including diagrams, tables, and export to Excel.

ado-core

A framework for designing, executing and analysing experiment campaigns

uptrain

UpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them.

mltraq

Track and Collaborate on ML & AI Experiments.

ado-vllm-performance

A framework for designing, executing and analysing experiment campaigns

gbstats

Open Source Feature Flags, Experimentation, and Product Analytics

pipelinex

PipelineX: Python package to build ML pipelines for experimentation with Kedro, MLflow, and more

feedx

Transparent, robust and trustworthy A/B experimentation for Shopping feeds.

expan

Open-source Python library for statistical analysis of randomised control trials (A/B tests)

ml-labs

Pipeline-centric ML framework for experiment, train, and inference workflows

moai-mdk

moai is a PyTorch-based AI Model Development Kit (MDK) created to improve data-driven model workflows, design and reproducibility.

dead-salmon-brain

Apache Spark based framework for analysis A/B experiments

agentrylab

Lightweight, hackable multi-agent orchestration lab (CLI + Python) with transcripts, checkpoints, budgets, and pluggable providers/tools.

ir-axioms

↕️ Intuitive axiomatic retrieval experimentation.

hyperparameter-hunter

Easy hyperparameter optimization and automatic result saving across machine learning algorithms and libraries

yuclid

Orchestrating experiments on N-dimensional parameter spaces with custom metric collection + sliced-space visualization

blogstrap

An opinionated blogging platform

experiment-design

Tools to create experiment designs

model-failure-lab

Toolkit for systematically probing, classifying, and debugging failure modes in LLM and RAG systems — reasoning errors, hallucination, and API-level behaviour.

hydra-ml

A cloud-agnostic ML Platform