Swe Bench Python Packages

repo2rlenv

Turn any repository into verifiable RL environments for coding agents - Harbor tasks you can train on, evaluate and share on the Hugging Face Hub

coder-eval

Playwright for coding agents. Test that your skills, MCP servers, and CLIs actually work when an agent uses them — sandboxed YAML suites, A/B experiments, CI gates.

otto-cli-agent

A terminal AI agent that works on a codebase, container, browser or desktop, uses what it built, and judges its work against criteria written before it started. One agent loop, rubric-first evaluator, tiered memory, four-vendor routing, six benchmark harnesses.

agent-sandbox-e2b

Fast, Multi-Cloud Sandbox Engine for AI Agents

awb

Benchmark harness measuring AI coding tool+workflow performance, not just model capability. 100 tasks, sigmoid scoring, 12 capability dimensions, gap analysis.

mcpbr

Benchmark your MCP server.

jittest

Differential test-execution gate for agent-authored pull requests. Emits Ed25519-signed receipts you can re-verify without trusting the tool.

agent-sandbox-harbor

Fast, Multi-Cloud Sandbox Engine for AI Agents

skeletongraph

Zero-LLM structural code retrieval for AI coding agents — tree-sitter index, BM25+dense+structural fusion, served over MCP.

misterdev

Autonomous LLM build orchestrator: plans a goal into tasks, edits code with anchored SEARCH/REPLACE, and verifies every change through build/test/lint/typecheck gates before reporting done. Runs as a CLI or MCP server.

benchmaxxing-detector

Forensic audit scanner for SWE-bench & DeepSWE v1.1 containers detecting .git reflog leakage and test harness spoofing.

refact

AI Agent that handles engineering tasks end-to-end: integrates with developers’ tools, plans, executes, and iterates until it achieves a successful result.

chimera-run

Open-source Python framework for building coding agents. Pick your provider, pick your tools, pick your loop. Chimera wires them together. Recreate SWE-Agent, Aider, Cline, Codex in a few lines, or design something entirely new.

tasksmith

Turn any repository into verifiable RL environments for coding agents - Harbor tasks you can train on, evaluate and share on the Hugging Face Hub

homefield

Benchmarks are away games. Rank coding agents on YOUR repo's git history, with honest statistics, a flake filter, and CLAUDE.md rule ablation. Real run: Haiku vs Sonnet on simonw/llm.

state-harness

Runtime safety net for LLM agents. Detects token spirals, kills doomed tasks early, tells you exactly why. Rust core, Python SDK. pip install state-harness

forgejudge

Open, always-on leaderboard + CI gate for autonomous coding agents — every patch sandboxed, every run traced, every regression fails the build. $0 stack.

xanther-xce

Context engine for coding agents. Indexes a repo into a multi-layer knowledge graph and serves precise architectural context over MCP. 78.2% on SWE-bench Verified at $0.22/instance. MIT.

provenant

Microsoft Build AI 2026 National 1st Place. Local codebase memory for AI coding agents: 63.8% Coverage@5, 75.2% Coverage@10, 60-65x lower context.

patchward-check

Evidence home for the verdict-layer approach: pre-registered, reproducible trust evaluation of AI-generated code changes - silent wrong ships 17/50 → 0/50.

emo-x-eval

EMO-X: adaptive, execution-based evaluation OS for AI agents.

squeez

Squeeze verbose LLM agent tool output down to only the relevant lines

traceverdict

Auditable state-regression evaluation for coding agents

scaffoldscope

Controlled coding-agent harness ablations with auditable traces, reproducible evidence bundles, and SWE-bench interoperability.