# BenchFlow documentation

> Run AI agents on benchmark tasks in sandboxes, score every attempt with the task's own tests, and keep everything they did.

Docs index for agents: https://www.benchflow.ai/docs/llms.txt

BenchFlow is a command-line tool, `bench`, and a Python package, `benchflow` (these pages cover 0.9.2), for evaluating AI agents. It runs an agent on each task in a fresh sandbox, scores what the agent left with the task's own checks, and keeps a record of each attempt that can explain its score: what the agent did, each model call through BenchFlow's proxy with the settings it carried, and the workspace it left. SkillsBench and Robo Use run on it.

## What do you want to do?

- Run my first eval: [Quick start](https://www.benchflow.ai/docs/benchflow/start/quick-start.md)
- See what that record lets you decide, where BenchFlow stops, and how its parts fit: [Why BenchFlow](https://www.benchflow.ai/docs/benchflow/start/why-benchflow.md), [How BenchFlow works](https://www.benchflow.ai/docs/benchflow/start/how-it-works.md)
- Measure an agent or a model on a benchmark: [Find a benchmark](https://www.benchflow.ai/docs/benchflow/evaluate/find-a-benchmark.md), then [Run a benchmark](https://www.benchflow.ai/docs/benchflow/evaluate/run-a-benchmark.md)
- Check a benchmark, or a published score: [Check a benchmark](https://www.benchflow.ai/docs/benchflow/audit/check-a-benchmark.md), [Reproduce a published score](https://www.benchflow.ai/docs/benchflow/audit/reproduce.md)
- Publish a score others can check: [Report a result](https://www.benchflow.ai/docs/benchflow/audit/report.md)
- Build tasks, or a whole benchmark: [Write a task](https://www.benchflow.ai/docs/benchflow/tasks/write-a-task.md)
- Grade design work by blind pairwise votes, from people or a model (0.10): [Human preferences](https://www.benchflow.ai/docs/benchflow/tasks/human-preferences.md)
- Score an agent I wrote: [Run your own agent](https://www.benchflow.ai/docs/benchflow/evaluate/your-own-agent.md)
- Run simulated robot tasks from several seeds: [Embodied tasks](https://www.benchflow.ai/docs/benchflow/evaluate/embodied.md)
- Train a model on scored attempts: [BenchFlow as an RL environment](https://www.benchflow.ai/docs/benchflow/train/environment.md)
- Run thousands of trials in the cloud: [Sandboxes](https://www.benchflow.ai/docs/benchflow/scale/sandboxes.md)
- Bring tasks, results or trajectories from another tool: [Bring what you have](https://www.benchflow.ai/docs/benchflow/evaluate/bring-what-you-have.md)
- Move a project from 0.7: [Upgrade from 0.7](https://www.benchflow.ai/docs/benchflow/versions/upgrade-from-0.7.md)

## Try it

You need [uv](https://docs.astral.sh/uv/getting-started/installation/) and Docker running. Install BenchFlow:

```bash
uv tool install --python 3.12 'benchflow==0.9.2' \
  --constraints https://raw.githubusercontent.com/benchflow-ai/benchflow/v0.9.2/constraints.txt
```

The constraints file pins every dependency to the version the release was tested with ([Install](https://www.benchflow.ai/docs/benchflow/start/install.md)).

Score a public task with its reference solution, the `oracle` agent, which needs no model and no key:

```bash
bench run benchflow-ai/skillsbench/tasks/econ-detrending-correlation -a oracle
```

It prints `Score: 1/1` when the solution passes the task's tests. The [Quick start](https://www.benchflow.ai/docs/benchflow/start/quick-start.md) then runs Claude Code on the same task and shows how to read what it did.

## Not in these guides

Two features of 0.9.2 have only their reference here: branching a trial from a saved sandbox snapshot ([`bench branch`](https://www.benchflow.ai/docs/benchflow/reference/commands.md#bench-branch) and `bench run --checkpoints`), and hosted environments ([`bench hub`](https://www.benchflow.ai/docs/benchflow/reference/commands.md#bench-hub) and `bench run --source-env`).

## Versions

These pages document BenchFlow 0.9.2, the current release. 0.9.2 is 0.9.1 with five grading-integrity fixes and nothing else ([Changelog](https://www.benchflow.ai/docs/benchflow/versions/changelog.md)). Outputs from runs on 0.9.1 say so, and hold for 0.9.2 unless a task's grading meets one of those fixes. Commands that download BenchFlow's example files name the repository's `main` branch in this Markdown. On the published site they fetch from `v0.10.0rc3` instead, the 0.10.0rc3 release tag, which holds these pages and their example files, so a page always runs against the files it was written with. The repository's Markdown and preview builds keep `main`. A page that needs 0.10.0rc3 says so in its first line, and the sidebar tags it `0.10`; it's on PyPI as a pre-release ([Install](https://www.benchflow.ai/docs/benchflow/start/install.md#install-the-010-prerelease)). A section that needs it ends its heading with "(0.10)". [What's in 0.10.0rc3](https://www.benchflow.ai/docs/benchflow/versions/0.10.md) lists them all.

## For coding agents

Every page is also Markdown: add `.md` to its URL, or send `Accept: text/markdown`. [`/docs/llms.txt`](https://www.benchflow.ai/docs/llms.txt) lists every page with its sections, and links one file per section with that section's pages in full.

Every page with its sections, for coding agents: https://www.benchflow.ai/docs/llms.txt

## All pages

### Start

- [Install](https://www.benchflow.ai/docs/benchflow/start/install.md): Install `bench` and the Python package at a pinned version, check the machine, and move between versions.
- [Quick start](https://www.benchflow.ai/docs/benchflow/start/quick-start.md): Score one task with its reference solution, with an agent that does nothing, and with Claude Code, then read the results.
- [How BenchFlow works](https://www.benchflow.ai/docs/benchflow/start/how-it-works.md): The rollout, BenchFlow's unit: what goes into one (task, environment, agent, model), what comes out (trajectory, reward, artifacts), and what reads them.
- [Why BenchFlow](https://www.benchflow.ai/docs/benchflow/start/why-benchflow.md): See what each rollout records, what BenchFlow decides from that record, and where it stops, against Han-chung Lee's posts.

### Evaluate an agent

- [Find a benchmark](https://www.benchflow.ai/docs/benchflow/evaluate/find-a-benchmark.md): See which benchmarks you can run: the registry, any folder of tasks in a Git repository, and your own folders.
- [Agents and models](https://www.benchflow.ai/docs/benchflow/evaluate/agents.md): Pick an agent (Claude Code, Codex, OpenCode, Pi and more), log in or set a key, and choose the model and effort.
- [Before you run an eval](https://www.benchflow.ai/docs/benchflow/evaluate/before-you-run.md): Settle the settings that change a score (harness, effort, sampling, tries, limits, controls, cost) and see the plan before anything runs.
- [Run a benchmark](https://www.benchflow.ai/docs/benchflow/evaluate/run-a-benchmark.md): Run an agent on many tasks at once, choose which tasks and how many tries, and know where the results go.
- [Read the results](https://www.benchflow.ai/docs/benchflow/evaluate/read-results.md): Read a job's score, each trial's outcome and reward, what the agent did and what it cost, from the shell, the browser or Python.
- [When a trial fails](https://www.benchflow.ai/docs/benchflow/evaluate/failed-trials.md): Find out why a trial failed or has no score, whose problem it is, and what to do next.
- [Skills](https://www.benchflow.ai/docs/benchflow/evaluate/skills.md): Give the agent skills, confirm it used them, and measure whether they help.
- [Run your own agent](https://www.benchflow.ai/docs/benchflow/evaluate/your-own-agent.md): Score an agent you wrote, over ACP or as a script, like any built-in agent.
- [Embodied tasks (Robo Use)](https://www.benchflow.ai/docs/benchflow/evaluate/embodied.md): Run Robo Use's simulated robot tasks from several seeds, read the results seed by seed, and give the reference solution its token.
- [Bring what you have](https://www.benchflow.ai/docs/benchflow/evaluate/bring-what-you-have.md): Run tasks in the Terminal-Bench (task.toml) format, convert or export them, read old BenchFlow jobs, and hand trajectories to tools that read ATIF; see what doesn't carry over yet.

### Audit and report

- [Check a benchmark](https://www.benchflow.ai/docs/benchflow/audit/check-a-benchmark.md): Before trusting a benchmark's scores, show that every task can be solved, none passes for free, the verifiers hold, and the tasks aren't flaky.
- [Compare runs](https://www.benchflow.ai/docs/benchflow/audit/compare-runs.md): Pick the metric, count failures the right way, and tell with an interval whether two runs really differ.
- [Reproduce a published score](https://www.benchflow.ai/docs/benchflow/audit/reproduce.md): Take a published number, match its settings, rerun it, and say whether it holds.
- [Report a result](https://www.benchflow.ai/docs/benchflow/audit/report.md): Publish a score with the settings, controls, intervals and files another researcher needs to check it.
- [Known pitfalls](https://www.benchflow.ai/docs/benchflow/audit/pitfalls.md): Problems that quietly change a score or break a run, each with its symptom, cause, how to detect it, and the fix.

### Build a benchmark

- [Write a task](https://www.benchflow.ai/docs/benchflow/tasks/write-a-task.md): Write a small research task (data in, a number out, a stated tolerance), check it, prove it with the controls, and run an agent on it.
- [Task format](https://www.benchflow.ai/docs/benchflow/tasks/task-format.md): The files of a task, what each is for, and the task.toml layout BenchFlow also reads; where both forms exist, they must match.
- [Multi-turn tasks](https://www.benchflow.ai/docs/benchflow/tasks/multi-turn.md): Give the agent several prompts in one sandbox: a scripted user that writes each next prompt, or several agents taking turns.
- [Multi-step tasks](https://www.benchflow.ai/docs/benchflow/tasks/multi-step.md): Split a task into steps that are each checked before the next one starts.
- [Verifiers](https://www.benchflow.ai/docs/benchflow/tasks/verifiers.md): Score a trial with a test script: pass or fail, partial credit, tolerances, and a separate verifier for long tasks.
- [Rubric review](https://www.benchflow.ai/docs/benchflow/tasks/rubric-review.md): Grade the work a test can't check, such as the method behind an answer, with a weighted rubric and a reviewer agent, and read its verdicts.
- [LLM judge](https://www.benchflow.ai/docs/benchflow/tasks/llm-judge.md): Grade what no script can check, a piece of writing or a web page, with a model and criteria you write, and know what the judge records.
- [Human preferences](https://www.benchflow.ai/docs/benchflow/tasks/human-preferences.md): Judge two outputs at a time, blind, live and with reasons, by a person or a model, then rank the agents with intervals.
- [Network and secrets](https://www.benchflow.ai/docs/benchflow/tasks/network-and-secrets.md): Control what the agent can reach, keep it off the web or away from given sites, and know which keys and files enter the sandbox.
- [Stop reward hacking](https://www.benchflow.ai/docs/benchflow/tasks/reward-hacking.md): Make sure an agent can't score without doing the work, and prove it with probes.
- [Publish a benchmark](https://www.benchflow.ai/docs/benchflow/tasks/publish.md): Share a folder of tasks through Git or the registry, version it, and declare the setup runs should match.

### Run at scale

- [Sandboxes](https://www.benchflow.ai/docs/benchflow/scale/sandboxes.md): Choose where trials run, and set up Docker, remote Docker or AgentCore.
- [Daytona](https://www.benchflow.ai/docs/benchflow/scale/daytona.md): Run trials in Daytona cloud sandboxes: setup, limits, and cleanup.
- [Modal](https://www.benchflow.ai/docs/benchflow/scale/modal.md): Run trials in Modal sandboxes, including tasks that need GPUs.
- [Concurrency and budgets](https://www.benchflow.ai/docs/benchflow/scale/cost.md): Run many trials at once, cap what a job may spend, and watch a long run.
- [Usage and telemetry](https://www.benchflow.ai/docs/benchflow/scale/telemetry.md): What BenchFlow records for every model call (tokens, cost, timing, settings), where each number comes from, and what happens when an agent exports no OpenTelemetry.
- [Resume, retries and regrade](https://www.benchflow.ai/docs/benchflow/scale/resume.md): Continue a stopped job, know which failures are retried, and rescore saved work with a fixed verifier.
- [Configs and variables](https://www.benchflow.ai/docs/benchflow/scale/configs.md): Keep a whole run in a YAML file, and send each environment variable where it's needed.
- [Run in CI](https://www.benchflow.ai/docs/benchflow/scale/ci.md): Run a benchmark in CI and fail the build when the pass rate drops.

### Train with RL

- [BenchFlow as an RL environment](https://www.benchflow.ai/docs/benchflow/train/environment.md): Run groups of rollouts on a task, score them, and read rewards and advantages, with token capture on from the start.
- [Export training data](https://www.benchflow.ai/docs/benchflow/train/export.md): Turn finished trials into rows a trainer reads, check the rows, and know each file a trial writes for trainers and other tools, with its schema.
- [Connect a trainer](https://www.benchflow.ai/docs/benchflow/train/trainer.md): Stream rollouts to a trainer while a job runs, check token capture, launch a Prime-RL SFT job, and connect other trainers through files.

### Versions

- [Upgrade from 0.7](https://www.benchflow.ai/docs/benchflow/versions/upgrade-from-0.7.md): Move an install and a project from 0.7.x to 0.9.2, and go back if you need to.
- [What's in 0.10.0rc3](https://www.benchflow.ai/docs/benchflow/versions/0.10.md): The prerelease's features, how to install it, and what may still change.
- [Changelog](https://www.benchflow.ai/docs/benchflow/versions/changelog.md): Every release's changes, from the repository's CHANGELOG.

### Reference

- [Commands](https://www.benchflow.ai/docs/benchflow/reference/commands.md): Look up every `bench` command and flag, including the ones `--help` hides and the ones new in 0.10.
- [Python API](https://www.benchflow.ai/docs/benchflow/reference/python.md): Look up every public name in the `benchflow` package.
- [CLI and Python](https://www.benchflow.ai/docs/benchflow/reference/cli-python.md): Find the Python argument that does what each `bench` flag does.
- [Files BenchFlow reads and writes](https://www.benchflow.ai/docs/benchflow/reference/files.md): Look up every key of the task files you write and every field of the job and trial files BenchFlow writes.
- [Error messages](https://www.benchflow.ai/docs/benchflow/reference/errors.md): Look up a fixed message BenchFlow prints and what to do about it.
- [Environment variables](https://www.benchflow.ai/docs/benchflow/reference/environment-variables.md): Every variable BenchFlow reads, and where it goes: your machine, the sandbox, the agent or the verifier.
- [Glossary](https://www.benchflow.ai/docs/benchflow/reference/glossary.md): Every term in these docs, one line each, with the page that covers it.