Inspect Ai Python Packages

hawk

Run Inspect AI evals in the cloud

agent-release-gates

Release gates for AI agents: replay known incidents and check prompt, model and tool-policy changes before they ship. Runs under Inspect.

aipsy-bench

Open-source psychological-safety benchmark for conversational AI: frozen clinical scenarios scored by an LLM-judge panel into a CI pass/fail gate + a clinician-grade diagnostic.

falsify-inspect

PRML pre-registration adapter for Inspect AI eval logs. Listed on the UK AI Safety Institute's Inspect extensions page. MIT.

agentsec-bench

Security benchmark for LLM agents & their harnesses — offensive, defensive, secure-coding & prompt-injection-safety challenges on live sandboxed targets, with gradated scoring. Built on Inspect AI.

ai-crucible

Diagnostic adversarial game for frontier LLMs — a policy-enforced kernel that mediates a Designer/Solver/Judge cycle, scores against a hidden oracle, and curates a Lab/Arena/Regression catalog.

truescore

Statistics for LLM-judged evaluations: bias-corrected scores, real confidence intervals, and comparisons that hold up

petri-braintrust

Export Inspect Petri alignment audits to Braintrust experiments, with first-class Ollama support.

wikigame-agent

LLM agent that plays the Wikipedia game, built on AISI Inspect

decision-contract-audit

An Inspect AI extension that audits an LLM decision point against six reliability contracts by replaying a decision log — keyless, no model calls, no API keys.

judgecal

Statistically rigorous, batch-first reliability auditing for LLM judges & reward models — clustered-bootstrap CIs, McNemar, BH-FDR, power/MDE, planted-bias validation. No API key required.

phionyx-eval-inspect

Inspect AI bridge — Phionyx runtime evidence exported into Inspect eval logs. Offline-inspectable evaluation records.