Each term these docs use, in a line, with the page that covers it.
| Term | Meaning | More |
|---|---|---|
| ACP | Agent Client Protocol: how BenchFlow talks to most agents. | Agents and models |
| Advantage | How much better a rollout did than the others in its group; trainers learn from it. | BenchFlow as an RL environment |
| Agent | The program that does a task, such as Claude Code or Codex. | Agents and models |
| Artifacts | Files a task declares, saved in each trial's folder and given to a separate verifier. | Task format |
| ATIF | A JSON format for agent trajectories; each trial writes one in trainer/atif.json. | Trajectory and rollout formats |
| Attempt | One try at a trial. A trial that fails for a reason another try could fix, such as a dropped connection, runs again in a new sandbox, up to 2 more times by default. The trial counts its scored attempt, or its newest when none was scored; a scored attempt is never retried. | Resume, retries and regrade |
| Benchmark | A set of tasks. Also called a dataset. | Find a benchmark |
| bfx | A small Linux program BenchFlow copies into each sandbox to run its own steps there; bench doctor checks it. | Install |
| Blocker | A rubric criterion that must pass; if it fails, the trial's reward is 0. | Rubric review |
| Control | A run whose result you know in advance: the oracle should score 1, nop 0. | Check a benchmark |
| Digest | A hash over a task's files; the same digest means the same task. | Publish a benchmark |
| Effort | How hard a reasoning model thinks, set with -e. | Agents and models |
empty | How the Control column of bench results names the nop agent's runs. | Read the results |
| Errored | A trial that stopped before it could be scored. | When a trial fails |
| Environment | What a task's sandbox is built from: its environment/Dockerfile and files, or a named image. | Task format |
| Group | Rollouts of the same task, agent and model, compared with each other for an advantage. | BenchFlow as an RL environment |
| Harness | The program around the model that does the task: the agent you pick with -a, such as Claude Code. The --harness flag sets only how BenchFlow launches it: over ACP (the default) or as its own CLI (native). | Agents and models |
| Integrity audit | --integrity audit: a verdict next to each reward on whether anything the agent did reached it. | Stop reward hacking |
| Job | The rollouts one command starts, one per task, with their folder under jobs/; Python calls it an Evaluation. | How BenchFlow works |
| LLM judge | A model that grades an answer against criteria you write, in place of a test. | LLM judge |
| Matrix | The jobs one command starts: the repeats of --trials, or one job per model listed in a --matrix file. Its totals print on the Matrix: line and in matrix-summary.json. | Run a benchmark |
| Model proxy | The proxy API-key model calls go through, which counts tokens and keeps your key from the agent; it runs on your machine for local Docker and inside each sandbox on Daytona and Modal. | Usage and telemetry |
| Multi-step task | A task in steps, each checked before the next starts (0.10). | Multi-step tasks |
| nop | The agent that does nothing; a control. | Check a benchmark |
| Oracle | The agent that runs a task's reference solution (oracle/solve.sh); a control. | Quick start |
| pass@k | The chance that at least one of k tries at a task passes. | Compare runs |
| pass^k | The chance that all k tries at a task pass. | Compare runs |
| Registry | The list of named, versioned datasets bench run -d resolves. | Find a benchmark |
| Repeat | One pass of a job over its whole task set. --trials 3 makes three repeats, each a job of its own in a trial-01 to trial-03 folder with one trial per task; the Matrix: line counts them as trial jobs. | Run a benchmark |
| Reward | The verifier's score for a trial, 0.0 to 1.0 unless the task widens it. | Verifiers |
| Rollout | One agent's attempt at one task, in its own sandbox: BenchFlow's unit, which every score, vote and training row is read from. The command line and the result folders call it a trial; Python calls it Rollout. | How BenchFlow works |
rollouts (dry run) | In 0.9.2's --dry-run plan, the number of tasks in one pass; --trials 3 runs three such passes. | Before you run an eval |
| Round | One exchange with a user: the agent works, a soft check runs, the user writes the next prompt or stops. | Multi-turn tasks |
| Rubric review | A reviewer agent grading a trial's work against weighted criteria. | Rubric review |
| Run card | The settings summary 0.10.0rc2 prints before a run. | Before you run an eval |
| Run config | A YAML file with a whole run's settings, given to bench run --config. | Configs and variables |
| Sandbox | The isolated machine a trial runs in. | Sandboxes |
| Scene | A list of turns by named roles in one sandbox; a task can run several in order. | Multi-turn tasks |
| Scored | A trial whose test reached a verdict: its reward, or 0 when the test itself broke. | When a trial fails |
| Script agent | A shell script run as the agent (-a script:./probe.sh), as probes of a verifier are. | Stop reward hacking |
| Scripted user | Python code that writes the agent's next prompt from the last round's result. | Multi-turn tasks |
| Separate verifier | The verifier run in a fresh sandbox that gets only the workspace and artifacts. | Verifiers |
| Skill | Instructions and files an agent can load for a kind of task. | Skills |
| Soft check | The task's test run between rounds or steps, on the workspace as the agent left it; the final check is the one that counts. | Multi-turn tasks |
| Task | A folder with an instruction, an environment, a verifier and usually a reference solution. | Task format |
| Trajectory | Everything the agent did in a trial: messages, thoughts, tool calls. | Read the results |
| Trial | The command line's and the result folders' name for a rollout. The --trials flag and the trial-NN folders count repeats instead (see Repeat). | How BenchFlow works |
| Turn | One prompt to one agent. | Multi-turn tasks |
| Unscored | A trial whose agent finished but whose verifier couldn't score it. | When a trial fails |
| Verifier | The task's check, usually verifier/test.sh, which writes the reward. | Verifiers |
| Warm pool | Sandboxes prepared ahead of their trial's turn on Docker: 1 by default, 0 with -j 1; --warm-pool sets the number. | Concurrency and budgets |
| Workspace | The folder the agent works in (the task's WORKDIR), which the verifier grades. | How BenchFlow works |