# BenchFlow docs

> BenchFlow runs AI agents on benchmark tasks in sandboxes, scores each attempt with the task's own tests, and exports the attempts for training.

Use it to measure an agent on a benchmark, to test a task you wrote, or to turn scored attempts into training data, in Docker on your machine or in cloud sandboxes.

This site documents BenchFlow 0.9.2. A page marked (0.10) needs 0.10.0rc3, a release candidate on PyPI that installs by its version, and says so in its first line; everything else runs on 0.9.2.

Use this index to choose the few pages you need, then fetch them: every link below is Markdown, and each line lists the page's sections. Each section of the docs also has its pages in one file, linked at its end. Every page in one file is https://www.benchflow.ai/docs/llms-full.txt (1368 KB); fetch it only to index the docs offline.

The same pages are HTML at the URL without `.md`, and a request with `Accept: text/markdown` gets the Markdown too.

## BenchFlow

Run AI agents on benchmark tasks in sandboxes, score every attempt with the task's own tests, and keep everything they did.

- [BenchFlow docs home](https://www.benchflow.ai/docs/benchflow.md): what BenchFlow is, what to do with it, and which page does each job.

### Start

- [Install](https://www.benchflow.ai/docs/benchflow/start/install.md): Install `bench` and the Python package at a pinned version, check the machine, and move between versions. Sections: Install; Check your machine; Add a cloud sandbox; Use it from Python; Install the 0.10 prerelease; Upgrade, go back or uninstall; Which bench runs.
- [Quick start](https://www.benchflow.ai/docs/benchflow/start/quick-start.md): Score one task with its reference solution, with an agent that does nothing, and with Claude Code, then read the results. Sections: Score it with the reference solution; Check that doing nothing scores 0; Run Claude Code on it; See what happened; Next.
- [How BenchFlow works](https://www.benchflow.ai/docs/benchflow/start/how-it-works.md): The rollout, BenchFlow's unit: what goes into one (task, environment, agent, model), what comes out (trajectory, reward, artifacts), and what reads them. Sections: What goes in, what comes out; The words; One rollout, step by step; What a rollout leaves on disk; Why the rollout; What BenchFlow does not do.
- [Why BenchFlow](https://www.benchflow.ai/docs/benchflow/start/why-benchflow.md): See what each rollout records, what BenchFlow decides from that record, and where it stops, against Han-chung Lee's posts. Sections: What each rollout records; What is decided from the record; Built for research benchmarks; Where it stops; Further reading.
- This section's pages in one file: https://www.benchflow.ai/docs/llms-full/benchflow/start.txt (38 KB)

### Evaluate an agent

- [Find a benchmark](https://www.benchflow.ai/docs/benchflow/evaluate/find-a-benchmark.md): See which benchmarks you can run: the registry, any folder of tasks in a Git repository, and your own folders. Sections: Browse the registry; Run a benchmark from a Git repository; Your own folder of tasks.
- [Agents and models](https://www.benchflow.ai/docs/benchflow/evaluate/agents.md): Pick an agent (Claude Code, Codex, OpenCode, Pi and more), log in or set a key, and choose the model and effort. Sections: Pick an agent; Log in or set a key; Choose the model and effort; A model from another provider; A model you serve; Pin the agent's version; Run Claude Code or Codex through their own CLI; Codex Apps.
- [Before you run an eval](https://www.benchflow.ai/docs/benchflow/evaluate/before-you-run.md): Settle the settings that change a score (harness, effort, sampling, tries, limits, controls, cost) and see the plan before anything runs. Sections: The settings that change a score; See the plan before it runs; What to write down; The run card (0.10).
- [Run a benchmark](https://www.benchflow.ai/docs/benchflow/evaluate/run-a-benchmark.md): Run an agent on many tasks at once, choose which tasks and how many tries, and know where the results go. Sections: Run three tasks, three times each; Choose the tasks; Repeat each task; Run many trials at once; Where results go; From Python.
- [Read the results](https://www.benchflow.ai/docs/benchflow/evaluate/read-results.md): Read a job's score, each trial's outcome and reward, what the agent did and what it cost, from the shell, the browser or Python. Sections: The summary; A job, trial by trial; Where the reward is; What the agent did; Tokens, cost and time; Everything in a trial's folder; In Python.
- [When a trial fails](https://www.benchflow.ai/docs/benchflow/evaluate/failed-trials.md): Find out why a trial failed or has no score, whose problem it is, and what to do next. Sections: Codes and what to do; When the test itself fails; Read the logs; Common failures; Keep the sandbox and look inside (0.10).
- [Skills](https://www.benchflow.ai/docs/benchflow/evaluate/skills.md): Give the agent skills, confirm it used them, and measure whether they help. Sections: Give the agent the task's skills; Confirm the agent used them; Measure whether skills help; Check a skill before you use it.
- [Run your own agent](https://www.benchflow.ai/docs/benchflow/evaluate/your-own-agent.md): Score an agent you wrote, over ACP or as a script, like any built-in agent. Sections: Choose how your agent talks to BenchFlow; An ACP agent; A script agent; Agents from the agents repository.
- [Embodied tasks (Robo Use)](https://www.benchflow.ai/docs/benchflow/evaluate/embodied.md): Run Robo Use's simulated robot tasks from several seeds, read the results seed by seed, and give the reference solution its token. Sections: Set it up; Run with seeds; Read the results by seed; The oracle token; Warnings you can ignore; Not covered here.
- [Bring what you have](https://www.benchflow.ai/docs/benchflow/evaluate/bring-what-you-have.md): Run tasks in the Terminal-Bench (task.toml) format, convert or export them, read old BenchFlow jobs, and hand trajectories to tools that read ATIF; see what doesn't carry over yet. Sections: Tasks in the task.toml format; Terminal-Bench 4.1; Old results; Trajectories and training rows; Model calls and OpenTelemetry; Take your tasks elsewhere.
- This section's pages in one file: https://www.benchflow.ai/docs/llms-full/benchflow/evaluate-an-agent.txt (97 KB)

### Audit and report

- [Check a benchmark](https://www.benchflow.ai/docs/benchflow/audit/check-a-benchmark.md): Before trusting a benchmark's scores, show that every task can be solved, none passes for free, the verifiers hold, and the tasks aren't flaky. Sections: Run the oracle and nop over every task; Read the control results; Terminal-Bench 2.0 on 0.9.1; Check for duplicate tasks and seen answers; Find flaky tasks; Check that a verifier accepts other correct answers; Probe the verifiers with a cheating agent; Search trajectories for shortcuts; Benchmarks with no reference solution.
- [Compare runs](https://www.benchflow.ai/docs/benchflow/audit/compare-runs.md): Pick the metric, count failures the right way, and tell with an interval whether two runs really differ. Sections: Repeat each task; Choose the metric; What counts in the denominator; Compare two runs task by task; How big a difference is real.
- [Reproduce a published score](https://www.benchflow.ai/docs/benchflow/audit/reproduce.md): Take a published number, match its settings, rerun it, and say whether it holds. Sections: Get the published trials; Write down the settings; Match them in a run; Compare with an interval; When the original harness differs.
- [Report a result](https://www.benchflow.ai/docs/benchflow/audit/report.md): Publish a score with the settings, controls, intervals and files another researcher needs to check it. Sections: What a result needs; Collect the evidence; Write it up; Share trajectories; The report file (0.10).
- [Known pitfalls](https://www.benchflow.ai/docs/benchflow/audit/pitfalls.md): Problems that quietly change a score or break a run, each with its symptom, cause, how to detect it, and the fix. Sections: Find your symptom; Settings that don't reach the model; Scores that count the wrong thing; Graders an agent can fool; Infrastructure that fails quietly; Many runs on one machine.
- This section's pages in one file: https://www.benchflow.ai/docs/llms-full/benchflow/audit-and-report.txt (72 KB)

### Build a benchmark

- [Write a task](https://www.benchflow.ai/docs/benchflow/tasks/write-a-task.md): Write a small research task (data in, a number out, a stated tolerance), check it, prove it with the controls, and run an agent on it. Sections: The task; Scaffold it; Build the environment and add the data; Write the instruction; Write the test; Write the reference solution; Check it and run the controls; Run an agent on it; Next.
- [Task format](https://www.benchflow.ai/docs/benchflow/tasks/task-format.md): The files of a task, what each is for, and the task.toml layout BenchFlow also reads; where both forms exist, they must match. Sections: The files; The task.md header; The environment; Services the agent works with; Other task formats.
- [Multi-turn tasks](https://www.benchflow.ai/docs/benchflow/tasks/multi-turn.md): Give the agent several prompts in one sandbox: a scripted user that writes each next prompt, or several agents taking turns. Sections: Turns and rounds; A scripted user; Several agents in one sandbox.
- [Multi-step tasks (0.10)](https://www.benchflow.ai/docs/benchflow/tasks/multi-step.md): Split a task into steps that are each checked before the next one starts. Sections: Declare the steps; Run it; How the reward is made.
- [Verifiers](https://www.benchflow.ai/docs/benchflow/tasks/verifiers.md): Score a trial with a test script: pass or fail, partial credit, tolerances, and a separate verifier for long tasks. Sections: Choose how a task is graded; How a verifier scores; Pass or fail; Partial credit and tolerances; Run the verifier in its own sandbox; When the verifier can't run; Fix a verifier and rescore.
- [Rubric review](https://www.benchflow.ai/docs/benchflow/tasks/rubric-review.md): Grade the work a test can't check, such as the method behind an answer, with a weighted rubric and a reviewer agent, and read its verdicts. Sections: When to use a rubric; Write the rubric; Run the review; Read the review; Check the reviewer; Choose the reviewer.
- [LLM judge](https://www.benchflow.ai/docs/benchflow/tasks/llm-judge.md): Grade what no script can check, a piece of writing or a web page, with a model and criteria you write, and know what the judge records. Sections: A model from another provider; Run the controls; What the judge records; Grade a web page; Limits in 0.9.2.
- [Human preferences (0.10)](https://www.benchflow.ai/docs/benchflow/tasks/human-preferences.md): Judge two outputs at a time, blind, live and with reasons, by a person or a model, then rank the agents with intervals. Sections: The design task; Collect the outputs; Make blind pairs; Judge each pair; Let a model judge the same packet; Import the votes; Check that every pair has its votes; Rank with intervals; Read the agreement between the judges; Outputs made outside BenchFlow; Votes as the rater's tool showed them; People, guests and model judges; and 4 more.
- [Network and secrets](https://www.benchflow.ai/docs/benchflow/tasks/network-and-secrets.md): Control what the agent can reach, keep it off the web or away from given sites, and know which keys and files enter the sandbox. Sections: What the agent can reach; Keep the agent off the web; Block given sites; What enters the sandbox; What the results keep.
- [Stop reward hacking](https://www.benchflow.ai/docs/benchflow/tasks/reward-hacking.md): Make sure an agent can't score without doing the work, and prove it with probes. Sections: What BenchFlow does between the agent and the verifier; What it can't stop; Probe your verifier; Audit how each reward was earned.
- [Publish a benchmark](https://www.benchflow.ai/docs/benchflow/tasks/publish.md): Share a folder of tasks through Git or the registry, version it, and declare the setup runs should match. Sections: Share through Git; Version the tasks; Add it to the registry; Publish in the task.toml layout too; Declare the setup runs should match (0.10).
- This section's pages in one file: https://www.benchflow.ai/docs/llms-full/benchflow/build-a-benchmark.txt (110 KB)

### Run at scale

- [Sandboxes](https://www.benchflow.ai/docs/benchflow/scale/sandboxes.md): Choose where trials run, and set up Docker, remote Docker or AgentCore. Sections: Choose a sandbox; Docker; Remote Docker; Amazon Bedrock AgentCore; What BenchFlow keeps between runs; Clean up.
- [Daytona](https://www.benchflow.ai/docs/benchflow/scale/daytona.md): Run trials in Daytona cloud sandboxes: setup, limits, and cleanup. Sections: Set it up; Run on Daytona; Limits; Clean up.
- [Modal](https://www.benchflow.ai/docs/benchflow/scale/modal.md): Run trials in Modal sandboxes, including tasks that need GPUs. Sections: Set it up; Run on Modal; GPUs; Limits.
- [Concurrency and budgets](https://www.benchflow.ai/docs/benchflow/scale/cost.md): Run many trials at once, cap what a job may spend, and watch a long run. Sections: Concurrency; Budgets; Watch a long run.
- [Usage and telemetry](https://www.benchflow.ai/docs/benchflow/scale/telemetry.md): What BenchFlow records for every model call (tokens, cost, timing, settings), where each number comes from, and what happens when an agent exports no OpenTelemetry. Sections: What a trial records; Where each number comes from; OpenTelemetry; Cost, and models without a price.
- [Resume, retries and regrade](https://www.benchflow.ai/docs/benchflow/scale/resume.md): Continue a stopped job, know which failures are retried, and rescore saved work with a fixed verifier. Sections: Resume a stopped job; Start fresh instead; Retries; Regrade saved work.
- [Configs and variables](https://www.benchflow.ai/docs/benchflow/scale/configs.md): Keep a whole run in a YAML file, and send each environment variable where it's needed. Sections: A run in a file; Environment variables.
- [Run in CI](https://www.benchflow.ai/docs/benchflow/scale/ci.md): Run a benchmark in CI and fail the build when the pass rate drops. Sections: Fail under a pass rate; Exit codes; A GitHub Actions job.
- This section's pages in one file: https://www.benchflow.ai/docs/llms-full/benchflow/run-at-scale.txt (44 KB)

### Train with RL

- [BenchFlow as an RL environment](https://www.benchflow.ai/docs/benchflow/train/environment.md): Run groups of rollouts on a task, score them, and read rewards and advantages, with token capture on from the start. Sections: The loop; Run a group of rollouts; Rewards and advantages; Check the rows before you train; From Python.
- [Export training data](https://www.benchflow.ai/docs/benchflow/train/export.md): Turn finished trials into rows a trainer reads, check the rows, and know each file a trial writes for trainers and other tools, with its schema. Sections: Make a job to export; Export a job; What a row holds; Check rows before training; Trajectory and rollout formats.
- [Connect a trainer](https://www.benchflow.ai/docs/benchflow/train/trainer.md): Stream rollouts to a trainer while a job runs, check token capture, launch a Prime-RL SFT job, and connect other trainers through files. Sections: Stream rollouts while a job runs; Groups and advantages; Token ids and logprobs; Train one step with TRL; Other trainers.
- This section's pages in one file: https://www.benchflow.ai/docs/llms-full/benchflow/train-with-rl.txt (27 KB)

### Versions

- [Upgrade from 0.7](https://www.benchflow.ai/docs/benchflow/versions/upgrade-from-0.7.md): Move an install and a project from 0.7.x to 0.9.2, and go back if you need to. Sections: Before you upgrade; Upgrade the install; Old jobs and resume; Scores count every trial; Changed commands; Changed tasks; Go back.
- [What's in 0.10.0rc3 (0.10)](https://www.benchflow.ai/docs/benchflow/versions/0.10.md): The prerelease's features, how to install it, and what may still change. Sections: Install the prerelease; What's new; Changes that can move a score or stop a task; What may still change.
- [Changelog](https://www.benchflow.ai/docs/benchflow/versions/changelog.md): Every release's changes, from the repository's CHANGELOG. Sections: [0.10.0rc3] - release candidate; [0.10.0rc2] - release candidate; [0.10.0rc1] - release candidate, not on PyPI; [0.9.2]; [0.9.1]; [0.9.0] - accepted, never published (0.9.1 replaces it); 0.7.11 (2026-10-16); 0.7.10 (2026-10-16); 0.7.9 (2026-10-14); 0.7.8 (2026-09-14); 0.7.7 (2026-09-09); 0.7.6 (2026-09-04); and 21 more.
- This section's pages in one file: https://www.benchflow.ai/docs/llms-full/benchflow/versions.txt (364 KB)

### Reference

- [Commands](https://www.benchflow.ai/docs/benchflow/reference/commands.md): Look up every `bench` command and flag, including the ones `--help` hides and the ones new in 0.10. Sections: bench; bench new; bench check; bench import; bench run; bench branch; bench results; bench view; bench regrade; bench export; bench doctor; bench agents; and 87 more.
- [Python API](https://www.benchflow.ai/docs/benchflow/reference/python.md): Look up every public name in the `benchflow` package. Sections: Functions; Classes; Values.
- [CLI and Python](https://www.benchflow.ai/docs/benchflow/reference/cli-python.md): Find the Python argument that does what each `bench` flag does. Sections: bench eval run; bench eval branch; bench eval metrics; bench eval compare; bench eval regrade; bench train convert; bench train stream.
- [Files BenchFlow reads and writes](https://www.benchflow.ai/docs/benchflow/reference/files.md): Look up every key of the task files you write and every field of the job and trial files BenchFlow writes. Sections: Files you write; Files BenchFlow writes.
- [Error messages](https://www.benchflow.ai/docs/benchflow/reference/errors.md): Look up a fixed message BenchFlow prints and what to do about it. Sections: Causes a run names; bench doctor warnings and failures; Errors that say what to do; Command errors; Problems in a task or config file; Tasks a sandbox can't run; What the verifier step reports; Warnings a run logs.
- [Environment variables](https://www.benchflow.ai/docs/benchflow/reference/environment-variables.md): Every variable BenchFlow reads, and where it goes: your machine, the sandbox, the agent or the verifier. Sections: Where a variable reaches; Model keys and logins; Let a task read your variables; BenchFlow's own variables.
- [Glossary](https://www.benchflow.ai/docs/benchflow/reference/glossary.md): Every term in these docs, one line each, with the page that covers it.
- This section's pages in one file: https://www.benchflow.ai/docs/llms-full/benchflow/reference.txt (598 KB)

## SkillsBench

The first benchmark for whether agents can use procedural skills.

- [SkillsBench docs home](https://www.benchflow.ai/docs/skillsbench.md): what SkillsBench is, what to do with it, and which page does each job.

### Get started

- [Getting Started](https://www.benchflow.ai/docs/skillsbench/getting-started.md): How to run SkillsBench — evaluate your coding agent's ability to use domain-specific skills. Sections: A single task; The full benchmark; Running from a remote repo (no clone needed); Running with skills; Validating a task; Tasks with Docker build failures; Tasks with intermittent oracle failures; Docker build failures; Timeout errors; ARM64 / Apple Silicon; API rate limiting.
- This section's pages in one file: https://www.benchflow.ai/docs/llms-full/skillsbench/get-started.txt (10 KB)

### Contribute

- [Contributing](https://www.benchflow.ai/docs/skillsbench/contributing.md): How to contribute tasks to SkillsBench — the first benchmark that tests whether agent skills can improve agent performance. Sections: Getting Access; Getting Started; Task Requirements; Workflow; Priority Skill Categories; Task Types to Create; Contributing; Skills Source; instruction.md style; Skills Documentation; BenchFlow Runtime.
- This section's pages in one file: https://www.benchflow.ai/docs/llms-full/skillsbench/contribute.txt (8 KB)