Confidence Intervals Python Packages

mapie

A scikit-learn-compatible library for estimating prediction intervals and controlling risks, based on conformal predictions.

crepes

Python package for conformal prediction

orbit-eval

You have a robot. orbit tells you what to do next, and how you will know it worked. Zero-dependency CLI for LeRobot datasets and evaluations: what the recording got wrong, what it never covered, whether a change survived the noise.

fieldtrial

Statistically rigorous real-world evaluation for robot policies. Plan how many rollouts you need, run them randomized and blinded from a phone at the robot, let fieldtrial switch LeRobot or openpi policies itself, and get a report that only says "better" when a pre-registered test does.

resample

Randomization-based inference in Python

errorbars

Error bars for LLM evals: clustered SEs, paired comparisons, leaderboards, and power analysis. Python + web calculator.

assevra

An open-source reliability scorecard for LLM agents — measure grounding, safety, PII, and task-completion against fixed thresholds with confidence intervals.

pyellipse

A Python library for creating and analyzing confidence ellipses, including Hotelling's T-squared ellipses for multivariate statistical analysis and data visualization.

hypothesize

Robust statistics in Python

doubt

Bringing back uncertainty to machine learning.

lcb-gate

Statistical pass/fail gating for flaky tests and LLM evals: Wilson lower-bound verdicts, paired A/B, pytest plugin, stdlib only

agentrial

Statistical evaluation framework for AI agents

duvida

🧐 Calculating exact and approximate confidence and information metrics for deep learning on general purpose and chemistry tasks.

kernel-quantile-regression

Kernel quantile regression

ci-methods-analyser

Analyse efficacy of your own confidence interval (CI) methods

binoculars

Confidence intervals for binomial proportions.

pyfeldmancousins

General-purpose code to compute Feldman-Cousins confidence intervals

binconfint

Python binomial confidence interval calculations - F-distribution method

fast-exp-analytics

Python toolkit for A/B and A/B/C test analysis, confidence intervals, MDE and duration planning, Excel reports, and chat summaries.

truescore

Statistics for LLM-judged evaluations: bias-corrected scores, real confidence intervals, and comparisons that hold up

binomcikit

A Python package for calculating confidence intervals in statistical analysis using binomial distributions.

foostrap

Fast bootstrap resampling in Python

cigate

Eval-gated CI/CD for AI products: block a merge when answer quality statistically regresses, per failure-mode axis, with the LLM-judge's bias corrected for.

replicas

Bootstrap confidence intervals for classifier metrics on PySpark.