Experimentation Python Packages

optimizely-sdk

Python SDK for Optimizely Feature Experimentation and Optimizely Full Stack (legacy).

gofeatureflag-python-provider

GO Feature Flag is a simple, complete and lightweight self-hosted cloud native feature flag solution 100% Open Source. 🎛️

rapidfireai

RapidFire AI: Rapid AI Customization from RAG to Fine-Tuning

moptipy

Implementation of metaheuristic optimization methods in Python for scientific, industrial, and educational scenarios. Experiments can be executed in parallel or in a distributed fashion. Experimental results can be evaluated in various ways, including diagrams, tables, and export to Excel.

ado-core

A framework for designing, executing and analysing experiment campaigns

uptrain

UpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them.

yuclid

Orchestrating experiments on N-dimensional parameter spaces with custom metric collection + sliced-space visualization

mltraq

Track and Collaborate on ML & AI Experiments.

ado-vllm-performance

A framework for designing, executing and analysing experiment campaigns

pipelinex

PipelineX: Python package to build ML pipelines for experimentation with Kedro, MLflow, and more

feedx

Package for running A/B experiments on shopping feeds.

ml-labs

No description available

moai-mdk

moai: Accelerated, Flexible, Modular, Reproducible, Insightful AI

expan

Open-source Python library for statistical analysis of randomised control trials (A/B tests)

blogstrap

An opinionated blogging platform

model-failure-lab

Toolkit for discovering, classifying, and debugging failure modes in LLM and RAG systems.

agentrylab

Lightweight, hackable multi-agent orchestration lab (CLI + Python) with transcripts, checkpoints, budgets, and pluggable providers/tools.

dead-salmon-brain

Apache Spark based framework for analysis A/B experiments

ir-axioms

↕️ Intuitive axiomatic retrieval experimentation.

gbstats

Open Source Feature Flags, Experimentation, and Product Analytics

hyperparameter-hunter

Easy hyperparameter optimization and automatic result saving across machine learning algorithms and libraries

hydra-ml

A cloud-agnostic ML Platform that will enable Data Scientists to run multiple experiments, perform hyper parameter optimization, evaluate results and serve models (batch/realtime) while still maintaining a uniform development UX across cloud environments

experiment-design

Tools to create designs of experiments

trialcheck

Audit completed A/B experiment readouts for SRM, peeking risk, practical significance, guardrail movement, and pre-period imbalance. Zero dependencies. Structured PASS/WARN/FAIL reports in JSON, Markdown, and HTML.