# What's in 0.10.0rc3

> The prerelease's features, how to install it, and what may still change.

Docs index for agents: https://www.benchflow.ai/docs/llms.txt

> 0.10.0rc3 is a release candidate, on PyPI as a pre-release. Its new commands and flags may still change before 0.10.0; use 0.9.2 for results you'll publish unless you need one of these.

## Install the prerelease

```bash
uv tool install --python 3.12 'benchflow==0.10.0rc3' \
  --constraints https://raw.githubusercontent.com/benchflow-ai/benchflow/v0.10.0rc3/constraints.txt
bench --version
```

```text
benchflow 0.10.0rc3
```

Without the version, uv and pip install 0.9.2 ([Install](https://www.benchflow.ai/docs/benchflow/start/install.md#install-the-010-prerelease)). To go back, install `'benchflow==0.9.2'` the same way. 0.10.0rc1 was built and tested but never published; 0.10.0rc2 has all of it, and 0.9.2's five grading-integrity fixes. 0.10.0rc3 is 0.10.0rc2 with the rows marked (rc3) below and the fixes in the [Changelog](https://www.benchflow.ai/docs/benchflow/versions/changelog.md).

## What's new

| Feature | What it does | Page |
|---|---|---|
| Run card | Before every run, a summary of the settings that change a score, with warnings; `--dry-run` prints it and stops; saved as `run-card.json` | [Before you run an eval](https://www.benchflow.ai/docs/benchflow/evaluate/before-you-run.md#the-run-card-010) |
| Run checks | Each trial's ending and model calls are checked against the card; a run whose trials were cut at the output cap, or whose calls carried other settings than it asked for, fails | [Before you run an eval](https://www.benchflow.ai/docs/benchflow/evaluate/before-you-run.md#the-run-checks) |
| One scoring rule | `benchflow.score.v1`: every finished trial counts, and only a scored pass counts 1; each score has a 95% interval over tasks | [Read the results](https://www.benchflow.ai/docs/benchflow/evaluate/read-results.md) |
| Per-trial budgets | `--token-budget` and `--turn-budget` cap each trial's tokens and model calls while it runs | [Concurrency and budgets](https://www.benchflow.ai/docs/benchflow/scale/cost.md#per-trial-budgets-010) |
| Reference setup | A task set's `run-reference.toml` declares the setup its numbers used; a strict one refuses runs that differ unless you give `--accept-deviations` | [Publish a benchmark](https://www.benchflow.ai/docs/benchflow/tasks/publish.md#declare-the-setup-runs-should-match-010) |
| Report file | Each job writes `report.md`, its settings and results in one file | [Report a result](https://www.benchflow.ai/docs/benchflow/audit/report.md#the-report-file-010) |
| Utilization lines | During a run, how many trial slots are busy and how much of the agents' time a model request is in flight, with a warning when the run is underused | [Concurrency and budgets](https://www.benchflow.ai/docs/benchflow/scale/cost.md#watch-a-long-run) |
| Keep and attach | `--keep-sandbox on-failure` keeps a failed trial's sandbox; `bench sandbox attach` opens a shell in it, `bench sandbox release` deletes it | [When a trial fails](https://www.benchflow.ai/docs/benchflow/evaluate/failed-trials.md#keep-the-sandbox-and-look-inside-010) |
| Multi-step tasks | `[[steps]]` in `task.toml`: each step checked before the next | [Multi-step tasks](https://www.benchflow.ai/docs/benchflow/tasks/multi-step.md) |
| Human preferences | `bench preferences pairs`, `import` and `rank`: blind pairs, votes from any rating tool, Bradley-Terry ratings with rater checks | [Human preferences](https://www.benchflow.ai/docs/benchflow/tasks/human-preferences.md) |
| Outside outputs and the vote site (rc3) | `pairs --items` pairs a golden or a remix made in another tool beside the trials; `import` reads the Frontier Design Benchmark vote site's export; `rank` rates people apart from guests and model judges, measures each judge against people, and resamples raters and tasks | [Human preferences](https://www.benchflow.ai/docs/benchflow/tasks/human-preferences.md#outputs-made-outside-benchflow) |
| Whole training groups (rc3) | `bench train stream --group-size N` passes a group on whole or not at all | [Connect a trainer](https://www.benchflow.ai/docs/benchflow/train/trainer.md) |
| Assessments | `bench assess`: one blind pair at a time for an external collector | [Human preferences](https://www.benchflow.ai/docs/benchflow/tasks/human-preferences.md#assessments-for-an-outside-collector) |
| Terminal-Bench 4.1 | `terminal-bench@4.1` in the registry, its 66 tasks pinned by digest, and the separate-verifier fixes that let its tasks be graded | [Bring what you have](https://www.benchflow.ai/docs/benchflow/evaluate/bring-what-you-have.md#terminal-bench-41) |
| OpenTelemetry | Each trial writes its trace as OTLP/JSON to `trajectory/otel-spans.json`, and sends it to the collector `OTEL_EXPORTER_OTLP_ENDPOINT` names | [Usage and telemetry](https://www.benchflow.ai/docs/benchflow/scale/telemetry.md#opentelemetry) |
| Trainer formats | `bench train convert --format` also writes `verl-sft`, `skyrl-sft` and `openenv-episodes`, and keeps a Claude Code trial's tool calls | [Export training data](https://www.benchflow.ai/docs/benchflow/train/export.md) |
| Regrade from saved outputs | `bench regrade` scores a trial that ran with a separate verifier again, and reruns its rubric review with `--review` | [Resume, retries and regrade](https://www.benchflow.ai/docs/benchflow/scale/resume.md) |
| Reviewer resources | `--reviewer-memory-mb`, `--reviewer-storage-mb`, `--reviewer-setup-timeout` and `--reviewer-evidence-timeout` for rubric reviews | [Commands](https://www.benchflow.ai/docs/benchflow/reference/commands.md#bench-run) |

Smaller changes: `bench run --help` lists `--trials`, `--integrity`, `--max-cost-usd` and `--max-tokens`, which 0.9.2 shows only under `--help-all`; `--dry-run` counts every trial of a `--trials` run, where 0.9.2 counts one pass; `--trials` repeats a task named by its address, which 0.9.2 refuses; Claude Code is 2.1.316, its default model `claude-haiku-5-5`, and it can use a model id its adapter doesn't list on a provider's own endpoint; BenchFlow carries vendors' own prices for 18 current models, MiMo V2.6 and DeepSeek V4 among them ([Usage and telemetry](https://www.benchflow.ai/docs/benchflow/scale/telemetry.md#cost-and-models-without-a-price)); an agent gets BenchFlow's variables as `RUNTIME_*` ([Run your own agent](https://www.benchflow.ai/docs/benchflow/evaluate/your-own-agent.md)); and embodied tasks also run on Daytona ([Embodied tasks](https://www.benchflow.ai/docs/benchflow/evaluate/embodied.md)). The full list is in the [Changelog](https://www.benchflow.ai/docs/benchflow/versions/changelog.md).

## Changes that can move a score or stop a task

These change what a 0.9.2 task scores or whether it runs, so a score from one version isn't comparable with the other's on the tasks they touch. From the [Changelog](https://www.benchflow.ai/docs/benchflow/versions/changelog.md):

| Change | 0.9.2 | 0.10.0rc3 | What to do |
|---|---|---|---|
| A repository's own `conftest.py` | removed before grading, so a correct answer whose tests need its fixtures scores 0 (SWE-bench Verified's Sphinx, Flask, pytest and seaborn tasks) | kept when it is byte for byte the pre-agent copy's and the agent ran as a user other than root; a separate verifier sandbox still removes it | rerun such tasks on 0.10.0rc3 before comparing ([Known pitfalls](https://www.benchflow.ai/docs/benchflow/audit/pitfalls.md#correct-answers-score-0-on-tasks-whose-tests-use-the-repositorys-own-conftestpy)) |
| Terminal-Bench 4.1 | not scored: none of the five tasks we ran got a score | graded: `interleaved-vigenere`'s reference solution scores 1 | report it from 0.10.0rc2 or later only, after the oracle on each task you use |
| A `task.toml` task whose separate verifier is built from `tests/Dockerfile` and declares artifacts | the verifier's workspace is emptied and gets the agent's whole workspace | keeps the files its image put there, and gets only the declared artifacts | declare every file the verifier reads from the agent's workspace |
| Claude Code with `--sandbox-user none`, or on a task that declares MCP servers | not refused | refused: claude.ai connectors and MCP servers are turned off, and it needs a sandbox user | run Claude Code as the sandbox user, and such a task with Codex or OpenCode |
| Codex asked for an effort the model doesn't offer | runs at `medium` | refused before the agent starts, with the efforts it offers | pass an effort the model offers |
| A named job (`--job`) whose settings or tasks changed | its finished trials are set aside and run again | refused, unless you pass `--restart-changed` | start a new job, or restart on purpose |
| The idle limit when the run sets none | 10 minutes | 30 minutes at effort `xhigh` or `max`, 10 otherwise | set `--agent-idle-timeout` when you compare across versions |
| An allowlist task's Compose services | may sit on any network | must sit on an `internal: true` network that `main` joins, or the trial stops before the agent starts | move the services to an internal network |
| An allowlist task with a compose file on `daytona` | not refused | refused before any sandbox starts | run it on `docker` or `remote-docker` |
| `network_mode: allowlist` with no `allowed_hosts` | refused when the task loads | admits only the agent's model endpoint | nothing; such a task now runs |

## What may still change

Commands and flags new in this release candidate can change name or shape before 0.10.0. What doesn't work yet in it: multi-step tasks can't use a separate verifier or per-step artifacts, and its open known issues are in the [Changelog](https://www.benchflow.ai/docs/benchflow/versions/changelog.md).