0.10.0rc3 is a release candidate, on PyPI as a pre-release. Its new commands and flags may still change before 0.10.0; use 0.9.2 for results you'll publish unless you need one of these.
Install the prerelease
uv tool install --python 3.12 'benchflow==0.10.0rc3' \
--constraints https://raw.githubusercontent.com/benchflow-ai/benchflow/v0.10.0rc3/constraints.txt
bench --versionbenchflow 0.10.0rc3Without the version, uv and pip install 0.9.2 (Install). To go back, install 'benchflow==0.9.2' the same way. 0.10.0rc1 was built and tested but never published; 0.10.0rc2 has all of it, and 0.9.2's five grading-integrity fixes. 0.10.0rc3 is 0.10.0rc2 with the rows marked (rc3) below and the fixes in the Changelog.
What's new
| Feature | What it does | Page |
|---|---|---|
| Run card | Before every run, a summary of the settings that change a score, with warnings; --dry-run prints it and stops; saved as run-card.json | Before you run an eval |
| Run checks | Each trial's ending and model calls are checked against the card; a run whose trials were cut at the output cap, or whose calls carried other settings than it asked for, fails | Before you run an eval |
| One scoring rule | benchflow.score.v1: every finished trial counts, and only a scored pass counts 1; each score has a 95% interval over tasks | Read the results |
| Per-trial budgets | --token-budget and --turn-budget cap each trial's tokens and model calls while it runs | Concurrency and budgets |
| Reference setup | A task set's run-reference.toml declares the setup its numbers used; a strict one refuses runs that differ unless you give --accept-deviations | Publish a benchmark |
| Report file | Each job writes report.md, its settings and results in one file | Report a result |
| Utilization lines | During a run, how many trial slots are busy and how much of the agents' time a model request is in flight, with a warning when the run is underused | Concurrency and budgets |
| Keep and attach | --keep-sandbox on-failure keeps a failed trial's sandbox; bench sandbox attach opens a shell in it, bench sandbox release deletes it | When a trial fails |
| Multi-step tasks | [[steps]] in task.toml: each step checked before the next | Multi-step tasks |
| Human preferences | bench preferences pairs, import and rank: blind pairs, votes from any rating tool, Bradley-Terry ratings with rater checks | Human preferences |
| Outside outputs and the vote site (rc3) | pairs --items pairs a golden or a remix made in another tool beside the trials; import reads the Frontier Design Benchmark vote site's export; rank rates people apart from guests and model judges, measures each judge against people, and resamples raters and tasks | Human preferences |
| Whole training groups (rc3) | bench train stream --group-size N passes a group on whole or not at all | Connect a trainer |
| Assessments | bench assess: one blind pair at a time for an external collector | Human preferences |
| Terminal-Bench 4.1 | terminal-bench@4.1 in the registry, its 66 tasks pinned by digest, and the separate-verifier fixes that let its tasks be graded | Bring what you have |
| OpenTelemetry | Each trial writes its trace as OTLP/JSON to trajectory/otel-spans.json, and sends it to the collector OTEL_EXPORTER_OTLP_ENDPOINT names | Usage and telemetry |
| Trainer formats | bench train convert --format also writes verl-sft, skyrl-sft and openenv-episodes, and keeps a Claude Code trial's tool calls | Export training data |
| Regrade from saved outputs | bench regrade scores a trial that ran with a separate verifier again, and reruns its rubric review with --review | Resume, retries and regrade |
| Reviewer resources | --reviewer-memory-mb, --reviewer-storage-mb, --reviewer-setup-timeout and --reviewer-evidence-timeout for rubric reviews | Commands |
Smaller changes: bench run --help lists --trials, --integrity, --max-cost-usd and --max-tokens, which 0.9.2 shows only under --help-all; --dry-run counts every trial of a --trials run, where 0.9.2 counts one pass; --trials repeats a task named by its address, which 0.9.2 refuses; Claude Code is 2.1.316, its default model claude-haiku-5-5, and it can use a model id its adapter doesn't list on a provider's own endpoint; BenchFlow carries vendors' own prices for 18 current models, MiMo V2.6 and DeepSeek V4 among them (Usage and telemetry); an agent gets BenchFlow's variables as RUNTIME_* (Run your own agent); and embodied tasks also run on Daytona (Embodied tasks). The full list is in the Changelog.
Changes that can move a score or stop a task
These change what a 0.9.2 task scores or whether it runs, so a score from one version isn't comparable with the other's on the tasks they touch. From the Changelog:
| Change | 0.9.2 | 0.10.0rc3 | What to do |
|---|---|---|---|
A repository's own conftest.py | removed before grading, so a correct answer whose tests need its fixtures scores 0 (SWE-bench Verified's Sphinx, Flask, pytest and seaborn tasks) | kept when it is byte for byte the pre-agent copy's and the agent ran as a user other than root; a separate verifier sandbox still removes it | rerun such tasks on 0.10.0rc3 before comparing (Known pitfalls) |
| Terminal-Bench 4.1 | not scored: none of the five tasks we ran got a score | graded: interleaved-vigenere's reference solution scores 1 | report it from 0.10.0rc2 or later only, after the oracle on each task you use |
A task.toml task whose separate verifier is built from tests/Dockerfile and declares artifacts | the verifier's workspace is emptied and gets the agent's whole workspace | keeps the files its image put there, and gets only the declared artifacts | declare every file the verifier reads from the agent's workspace |
Claude Code with --sandbox-user none, or on a task that declares MCP servers | not refused | refused: claude.ai connectors and MCP servers are turned off, and it needs a sandbox user | run Claude Code as the sandbox user, and such a task with Codex or OpenCode |
| Codex asked for an effort the model doesn't offer | runs at medium | refused before the agent starts, with the efforts it offers | pass an effort the model offers |
A named job (--job) whose settings or tasks changed | its finished trials are set aside and run again | refused, unless you pass --restart-changed | start a new job, or restart on purpose |
| The idle limit when the run sets none | 10 minutes | 30 minutes at effort xhigh or max, 10 otherwise | set --agent-idle-timeout when you compare across versions |
| An allowlist task's Compose services | may sit on any network | must sit on an internal: true network that main joins, or the trial stops before the agent starts | move the services to an internal network |
An allowlist task with a compose file on daytona | not refused | refused before any sandbox starts | run it on docker or remote-docker |
network_mode: allowlist with no allowed_hosts | refused when the task loads | admits only the agent's model endpoint | nothing; such a task now runs |
What may still change
Commands and flags new in this release candidate can change name or shape before 0.10.0. What doesn't work yet in it: multi-step tasks can't use a separate verifier or per-step artifacts, and its open known issues are in the Changelog.