Interpretability Python Packages

shap

A game theoretic approach to explain the output of any machine learning model.

interpret-core

Fit interpretable models. Explain blackbox machine learning.

tensorflow-decision-forests

A collection of state-of-the-art algorithms for the training, serving and interpretation of Decision Forest models in Keras.

captum

Model interpretability and understanding for PyTorch

interpret

Fit interpretable models. Explain blackbox machine learning.

ydf

A library to train, evaluate, interpret, and productionize decision forest models such as Random Forest and Gradient Boosted Decision Trees.

aplr

APLR builds predictive, interpretable regression and classification models using Automatic Piecewise Linear Regression. It often rivals tree-based methods in predictive accuracy while offering smoother and interpretable predictions.

grad-cam

Advanced AI Explainability for computer vision. Support for CNNs, Vision Transformers, Classification, Object detection, Segmentation, Image similarity and more.

raiutils

Responsible AI Toolbox is a suite of tools providing model and data exploration and assessment user interfaces and libraries that enable a better understanding of AI systems. These interfaces and libraries empower developers and stakeholders of AI systems to develop and monitor AI more responsibly, and take better data-driven actions.

imodels

Interpretable ML package 🔍 for concise, transparent, and accurate predictive modeling (sklearn-compatible).

dalex

moDel Agnostic Language for Exploration and eXplanation

nnsight

The nnsight package enables interpreting and manipulating the internals of deep learned models.

shapiq

Shapley Interactions and Shapley Values for Machine Learning

erroranalysis

Responsible AI Toolbox is a suite of tools providing model and data exploration and assessment user interfaces and libraries that enable a better understanding of AI systems. These interfaces and libraries empower developers and stakeholders of AI systems to develop and monitor AI more responsibly, and take better data-driven actions.

alibi

Algorithms for explaining machine learning models

fasttreeshap

Fast SHAP value computation for interpreting tree-based models

responsibleai

Responsible AI Toolbox is a suite of tools providing model and data exploration and assessment user interfaces and libraries that enable a better understanding of AI systems. These interfaces and libraries empower developers and stakeholders of AI systems to develop and monitor AI more responsibly, and take better data-driven actions.

sage-importance

For calculating global feature importance using Shapley values.

styxx

The measurement layer for machine minds. Reads what a model means and whether it holds the truth; certifies every claim re-runs. meaning_diff + OATH certify + mind profiles + live grounding signal + the cognometric instruments. No torch, no LLM in the loop for the core; MIT, open at the core.

gamma-facet

Human-explainable AI.

torchcam

Class activation maps for your PyTorch models (CAM, Grad-CAM, Grad-CAM++, Smooth Grad-CAM++, Score-CAM, SS-CAM, IS-CAM, XGrad-CAM, Layer-CAM)

shapash

🔅 Shapash: User-friendly Explainability and Interpretability to Develop Reliable and Transparent Machine Learning Models

pyvene

Stanford NLP Python library for understanding and improving PyTorch models via interventions

bnnr

XAI-driven augmentation & diagnostics for PyTorch vision - find model failures, fix with saliency-guided augmentation (ICD/AICD), prove with auditable reports.