BenchFlow documentation

BenchFlow is a command-line tool, bench, and a Python package, benchflow (these pages cover 0.9.2), for evaluating AI agents. It runs an agent on each task in a fresh sandbox, scores what the agent left with the task's own checks, and keeps a record of each attempt that can explain its score: what the agent did, each model call through BenchFlow's proxy with the settings it carried, and the workspace it left. SkillsBench and Robo Use run on it.

What do you want to do?

• Run my first eval: Quick start
• See what that record lets you decide, where BenchFlow stops, and how its parts fit: Why BenchFlow, How BenchFlow works
• Measure an agent or a model on a benchmark: Find a benchmark, then Run a benchmark
• Check a benchmark, or a published score: Check a benchmark, Reproduce a published score
• Publish a score others can check: Report a result
• Build tasks, or a whole benchmark: Write a task
• Grade design work by blind pairwise votes, from people or a model (0.10): Human preferences
• Score an agent I wrote: Run your own agent
• Run simulated robot tasks from several seeds: Embodied tasks
• Train a model on scored attempts: BenchFlow as an RL environment
• Run thousands of trials in the cloud: Sandboxes
• Bring tasks, results or trajectories from another tool: Bring what you have
• Move a project from 0.7: Upgrade from 0.7

Try it

You need uv and Docker running. Install BenchFlow:

uv tool install --python 3.12 'benchflow==0.9.2' \
  --constraints https://raw.githubusercontent.com/benchflow-ai/benchflow/v0.9.2/constraints.txt

The constraints file pins every dependency to the version the release was tested with (Install).

Score a public task with its reference solution, the oracle agent, which needs no model and no key:

bench run benchflow-ai/skillsbench/tasks/econ-detrending-correlation -a oracle

It prints Score: 1/1 when the solution passes the task's tests. The Quick start then runs Claude Code on the same task and shows how to read what it did.

Not in these guides

Two features of 0.9.2 have only their reference here: branching a trial from a saved sandbox snapshot (bench branch and bench run --checkpoints), and hosted environments (bench hub and bench run --source-env).

Versions

These pages document BenchFlow 0.9.2, the current release. 0.9.2 is 0.9.1 with five grading-integrity fixes and nothing else (Changelog). Outputs from runs on 0.9.1 say so, and hold for 0.9.2 unless a task's grading meets one of those fixes. Commands that download BenchFlow's example files name the repository's main branch in this Markdown. On the published site they fetch from v0.10.0rc3 instead, the 0.10.0rc3 release tag, which holds these pages and their example files, so a page always runs against the files it was written with. The repository's Markdown and preview builds keep main. A page that needs 0.10.0rc3 says so in its first line, and the sidebar tags it 0.10; it's on PyPI as a pre-release (Install). A section that needs it ends its heading with "(0.10)". What's in 0.10.0rc3 lists them all.

For coding agents

Every page is also Markdown: add .md to its URL, or send Accept: text/markdown. /docs/llms.txt lists every page with its sections, and links one file per section with that section's pages in full.