How BenchFlow works

BenchFlow's unit of work is the rollout: one agent, driving one model, works on one task in a fresh sandbox built from the task's environment, and when the agent stops, the task's verifier scores what it left. Everything else BenchFlow gives you is read from rollouts: a job's score, a comparison of two agents, a ranking from human votes, rows for a trainer. This page names the parts, then follows one real rollout through them: the Quick start's Claude Code run on MiMo, through BenchFlow's model proxy.

The command line and the result folders call a rollout a trial and a set of rollouts a job. In Python they are Rollout and Evaluation. The other pages say trial wherever you'll see that word in the output.

What goes in, what comes out

flowchart TB
  T["Task"] --> R(("Rollout"))
  E["Environment"] --> R
  A["Agent"] --> R
  M["Model"] --> R
  R --> TR["Trajectory"]
  R --> RW["Reward"]
  R --> AR["Artifacts"]
  TR --> V["Evaluation, preferences, training"]
  RW --> V
  AR --> V

A task, its environment, an agent and a model go in; a trajectory, a reward and the agent's files come out. Every other part of BenchFlow either sets up rollouts or reads what they left:

The words

One rollout, step by step

1. Start the sandbox. BenchFlow builds or pulls the task's image and starts a fresh sandbox from it.
2. Set up the agent. It installs the agent and gives it a user of its own (agent, not root). With an API key, it also starts a model proxy, on your machine for local Docker and inside the sandbox on Daytona and Modal; every model call goes through it, so BenchFlow can count tokens and record each request.
3. The agent's turn. The agent gets the task's instruction and works until it says it's done, or a time limit stops it.
4. Stop the agent and protect the checks. BenchFlow stops every process the agent left running and removes files it could use to fake a test run, so the agent can't change how it's graded after its turn.
5. Verify. BenchFlow copies the task's tests into the sandbox, runs test.sh, and reads the reward it writes.
6. Save and clean up. It writes the rollout's results, deletes the sandbox, and moves on to the next rollout.

The rollout's timing.json records how long each step took. For the quick start's Claude Code rollout on MiMo:

{
  "slot_wait": 0.0,
  "environment_setup": 2.1,
  "install_agent": 0.068,
  "sandbox_ready": 7.859,
  "proxy_health_wait": 0.0,
  "firewall_power": 0.099,
  "cloud_metadata_block": 0.421,
  "acp_connect": 1.877,
  "agent_setup": 2.4,
  "agent_execution": 372.3,
  "harden_before_verify": {
    "kill_sandbox_user_procs": 0.056,
    "clear_verifier_output": 0.01,
    "prepare_app_dir": 0.0,
    "reclaim_disk": 0.02,
    "freeze_workspace": 0.28,
    "cleanup_conftests": 0.596,
    "build_verifier_env": 0.603,
    "total": 1.85
  },
  "verifier": 5.3,
  "agent_stop_to_reward": 5.517,
  "provider_wait": 360.0,
  "provider_retry_wait": 0.0,
  "agent_without_provider_wait": 12.2,
  "cleanup": 0.383,
  "total": 388.6
}

The agent's turn took 372 seconds, and 360 of them were spent waiting for the model to answer. That split matters when a run is slow: here the model was the bottleneck, not the sandbox or the agent's tools. Each time is rounded on its own, so a difference can be 0.1 off (372.3 less 360.0 shows as 12.2), and some steps overlap, so they don't add up to total.

What a rollout leaves on disk

jobs/2026-11-04__16-43-38/
├── summary.json                      the score, counts and settings for the whole job
├── results.jsonl                     one line per rollout
└── econ-detrending-correlation__c6772b40/
    ├── result.json                   the reward (rewards.reward), outcome, errors, tokens
    ├── timing.json                   seconds per step
    ├── config.json                   the rollout's settings and BenchFlow version
    ├── trajectory/
    │   ├── acp_trajectory.jsonl      the agent's messages, thoughts and tool calls
    │   └── llm_trajectory.jsonl      each model request and response the proxy saw
    ├── verifier/
    │   ├── test-stdout.txt           what the tests printed
    │   └── reward.txt                the reward the verifier wrote
    ├── agent/                        the agent's own logs
    └── trainer/                      the same rollout in training formats

Every file in a rollout's folder, and every key of its result.json, is in Read the results; the files you write and the schema files BenchFlow writes are in Files BenchFlow reads and writes.

Why the rollout

BenchFlow made the rollout its core unit in v0.4.0, in May 2026, when the Python names Trial and Job became Rollout and Evaluation, terms the change took from Han-chung Lee's taxonomy of RL environments. Han-chung Lee, a SkillsBench author, gives the reason in his essay on agent evaluation infrastructure: "Rollouts is the core artifact of an Agentic AI system." An agent does its work by changing its sandbox, so its final answer alone can't show how it got there. A rollout keeps the whole attempt, and the same record serves a score, a vote and a training row. Why BenchFlow follows the rest of his argument.

What BenchFlow does not do

• It doesn't serve models. Agents call the model provider you choose.
• It doesn't train models. It turns rollouts into training data and hands them to a trainer: see Train with RL.
• It doesn't write tasks for you. It runs, checks and grades the tasks you or a benchmark provide.