Experimentation Python Packages

optimizely-sdk

Python SDK for Optimizely Feature Experimentation and Optimizely Full Stack (legacy).

gofeatureflag-python-provider

GO Feature Flag is a simple, complete and lightweight self-hosted cloud native feature flag solution 100% Open Source — built on OpenFeature 🎛️

gnosislab

A scientific-method runtime for AI agents: hypotheses, experiments, evidence, provenance, reproducibility, and auditable conclusions.

moptipy

Implementation of metaheuristic optimization methods in Python for scientific, industrial, and educational scenarios. Experiments can be executed in parallel or in a distributed fashion. Experimental results can be evaluated in various ways, including diagrams, tables, and export to Excel.

fieldtrial

Statistically rigorous real-world evaluation for robot policies. Plan how many rollouts you need, run them randomized and blinded from a phone at the robot, let fieldtrial switch LeRobot or openpi policies itself, and get a report that only says "better" when a pre-registered test does.

ado-core

A framework for designing, executing and analysing experiment campaigns

ado-vllm-performance

A framework for designing, executing and analysing experiment campaigns

ir-axioms

↕️ Intuitive axiomatic retrieval experimentation.

gbstats

Open Source Feature Flags, Experimentation, and Product Analytics

rapidfireai

RapidFire AI: Rapid AI Customization from RAG to Fine-Tuning

orts

Odds-Ratio Thompson Sampling: bandits and A/B tests whose baseline keeps moving. Paper: arXiv 2609.19709

experiment-utils-pd

Generic functions for experiment analysis and design

uptrain

UpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them.

yuclid

Orchestrating and visualizing experiments in N-dimensional irregular parameter spaces

mltraq

Track and Collaborate on ML & AI Experiments.

dead-salmon-brain

Apache Spark based framework for analysis A/B experiments

gallop-pds

🐎 Agentic riding for product data scientists.

noisefloor

Source-aware research diagnostics, correlation spectra, forecasting and synthetic Dyson experiments. Python, CLI, REST and MCP.

pipelinex

PipelineX: Python package to build ML pipelines for experimentation with Kedro, MLflow, and more

feedx

Transparent, robust and trustworthy A/B experimentation for Shopping feeds.

blogstrap

An opinionated blogging platform

moai-mdk

moai is a PyTorch-based AI Model Development Kit (MDK) created to improve data-driven model workflows, design and reproducibility.

increment

Python library for warehouse-native A/B testing, causal inference, and experiment analysis

expan

Open-source Python library for statistical analysis of randomised control trials (A/B tests)