# Glossary

> Every term in these docs, one line each, with the page that covers it.

Docs index for agents: https://www.benchflow.ai/docs/llms.txt

Each term these docs use, in a line, with the page that covers it.

| Term | Meaning | More |
|---|---|---|
| ACP | Agent Client Protocol: how BenchFlow talks to most agents. | [Agents and models](https://www.benchflow.ai/docs/benchflow/evaluate/agents.md) |
| Advantage | How much better a rollout did than the others in its group; trainers learn from it. | [BenchFlow as an RL environment](https://www.benchflow.ai/docs/benchflow/train/environment.md#rewards-and-advantages) |
| Agent | The program that does a task, such as Claude Code or Codex. | [Agents and models](https://www.benchflow.ai/docs/benchflow/evaluate/agents.md) |
| Artifacts | Files a task declares, saved in each trial's folder and given to a separate verifier. | [Task format](https://www.benchflow.ai/docs/benchflow/tasks/task-format.md) |
| ATIF | A JSON format for agent trajectories; each trial writes one in `trainer/atif.json`. | [Trajectory and rollout formats](https://www.benchflow.ai/docs/benchflow/train/export.md#trajectory-and-rollout-formats) |
| Attempt | One try at a trial. A trial that fails for a reason another try could fix, such as a dropped connection, runs again in a new sandbox, up to 2 more times by default. The trial counts its scored attempt, or its newest when none was scored; a scored attempt is never retried. | [Resume, retries and regrade](https://www.benchflow.ai/docs/benchflow/scale/resume.md#retries) |
| Benchmark | A set of tasks. Also called a dataset. | [Find a benchmark](https://www.benchflow.ai/docs/benchflow/evaluate/find-a-benchmark.md) |
| bfx | A small Linux program BenchFlow copies into each sandbox to run its own steps there; `bench doctor` checks it. | [Install](https://www.benchflow.ai/docs/benchflow/start/install.md#check-your-machine) |
| Blocker | A rubric criterion that must pass; if it fails, the trial's reward is 0. | [Rubric review](https://www.benchflow.ai/docs/benchflow/tasks/rubric-review.md#write-the-rubric) |
| Control | A run whose result you know in advance: the oracle should score 1, `nop` 0. | [Check a benchmark](https://www.benchflow.ai/docs/benchflow/audit/check-a-benchmark.md) |
| Digest | A hash over a task's files; the same digest means the same task. | [Publish a benchmark](https://www.benchflow.ai/docs/benchflow/tasks/publish.md#version-the-tasks) |
| Effort | How hard a reasoning model thinks, set with `-e`. | [Agents and models](https://www.benchflow.ai/docs/benchflow/evaluate/agents.md#choose-the-model-and-effort) |
| `empty` | How the Control column of `bench results` names the `nop` agent's runs. | [Read the results](https://www.benchflow.ai/docs/benchflow/evaluate/read-results.md) |
| Errored | A trial that stopped before it could be scored. | [When a trial fails](https://www.benchflow.ai/docs/benchflow/evaluate/failed-trials.md) |
| Environment | What a task's sandbox is built from: its `environment/Dockerfile` and files, or a named image. | [Task format](https://www.benchflow.ai/docs/benchflow/tasks/task-format.md#the-environment) |
| Group | Rollouts of the same task, agent and model, compared with each other for an advantage. | [BenchFlow as an RL environment](https://www.benchflow.ai/docs/benchflow/train/environment.md#rewards-and-advantages) |
| Harness | The program around the model that does the task: the agent you pick with `-a`, such as Claude Code. The `--harness` flag sets only how BenchFlow launches it: over ACP (the default) or as its own CLI (`native`). | [Agents and models](https://www.benchflow.ai/docs/benchflow/evaluate/agents.md) |
| Integrity audit | `--integrity audit`: a verdict next to each reward on whether anything the agent did reached it. | [Stop reward hacking](https://www.benchflow.ai/docs/benchflow/tasks/reward-hacking.md#audit-how-each-reward-was-earned) |
| Job | The rollouts one command starts, one per task, with their folder under `jobs/`; Python calls it an `Evaluation`. | [How BenchFlow works](https://www.benchflow.ai/docs/benchflow/start/how-it-works.md) |
| LLM judge | A model that grades an answer against criteria you write, in place of a test. | [LLM judge](https://www.benchflow.ai/docs/benchflow/tasks/llm-judge.md) |
| Matrix | The jobs one command starts: the repeats of `--trials`, or one job per model listed in a `--matrix` file. Its totals print on the `Matrix:` line and in `matrix-summary.json`. | [Run a benchmark](https://www.benchflow.ai/docs/benchflow/evaluate/run-a-benchmark.md#run-three-tasks-three-times-each) |
| Model proxy | The proxy API-key model calls go through, which counts tokens and keeps your key from the agent; it runs on your machine for local Docker and inside each sandbox on Daytona and Modal. | [Usage and telemetry](https://www.benchflow.ai/docs/benchflow/scale/telemetry.md) |
| Multi-step task | A task in steps, each checked before the next starts (0.10). | [Multi-step tasks](https://www.benchflow.ai/docs/benchflow/tasks/multi-step.md) |
| nop | The agent that does nothing; a control. | [Check a benchmark](https://www.benchflow.ai/docs/benchflow/audit/check-a-benchmark.md) |
| Oracle | The agent that runs a task's reference solution (`oracle/solve.sh`); a control. | [Quick start](https://www.benchflow.ai/docs/benchflow/start/quick-start.md) |
| pass@k | The chance that at least one of k tries at a task passes. | [Compare runs](https://www.benchflow.ai/docs/benchflow/audit/compare-runs.md) |
| pass^k | The chance that all k tries at a task pass. | [Compare runs](https://www.benchflow.ai/docs/benchflow/audit/compare-runs.md#choose-the-metric) |
| Registry | The list of named, versioned datasets `bench run -d` resolves. | [Find a benchmark](https://www.benchflow.ai/docs/benchflow/evaluate/find-a-benchmark.md) |
| Repeat | One pass of a job over its whole task set. `--trials 3` makes three repeats, each a job of its own in a `trial-01` to `trial-03` folder with one trial per task; the `Matrix:` line counts them as trial jobs. | [Run a benchmark](https://www.benchflow.ai/docs/benchflow/evaluate/run-a-benchmark.md#repeat-each-task) |
| Reward | The verifier's score for a trial, 0.0 to 1.0 unless the task widens it. | [Verifiers](https://www.benchflow.ai/docs/benchflow/tasks/verifiers.md) |
| Rollout | One agent's attempt at one task, in its own sandbox: BenchFlow's unit, which every score, vote and training row is read from. The command line and the result folders call it a trial; Python calls it `Rollout`. | [How BenchFlow works](https://www.benchflow.ai/docs/benchflow/start/how-it-works.md) |
| `rollouts` (dry run) | In 0.9.2's `--dry-run` plan, the number of tasks in one pass; `--trials 3` runs three such passes. | [Before you run an eval](https://www.benchflow.ai/docs/benchflow/evaluate/before-you-run.md#see-the-plan-before-it-runs) |
| Round | One exchange with a user: the agent works, a soft check runs, the user writes the next prompt or stops. | [Multi-turn tasks](https://www.benchflow.ai/docs/benchflow/tasks/multi-turn.md#turns-and-rounds) |
| Rubric review | A reviewer agent grading a trial's work against weighted criteria. | [Rubric review](https://www.benchflow.ai/docs/benchflow/tasks/rubric-review.md) |
| Run card | The settings summary 0.10.0rc2 prints before a run. | [Before you run an eval](https://www.benchflow.ai/docs/benchflow/evaluate/before-you-run.md#the-run-card-010) |
| Run config | A YAML file with a whole run's settings, given to `bench run --config`. | [Configs and variables](https://www.benchflow.ai/docs/benchflow/scale/configs.md) |
| Sandbox | The isolated machine a trial runs in. | [Sandboxes](https://www.benchflow.ai/docs/benchflow/scale/sandboxes.md) |
| Scene | A list of turns by named roles in one sandbox; a task can run several in order. | [Multi-turn tasks](https://www.benchflow.ai/docs/benchflow/tasks/multi-turn.md#several-agents-in-one-sandbox) |
| Scored | A trial whose test reached a verdict: its reward, or 0 when the test itself broke. | [When a trial fails](https://www.benchflow.ai/docs/benchflow/evaluate/failed-trials.md) |
| Script agent | A shell script run as the agent (`-a script:./probe.sh`), as probes of a verifier are. | [Stop reward hacking](https://www.benchflow.ai/docs/benchflow/tasks/reward-hacking.md#probe-your-verifier) |
| Scripted user | Python code that writes the agent's next prompt from the last round's result. | [Multi-turn tasks](https://www.benchflow.ai/docs/benchflow/tasks/multi-turn.md) |
| Separate verifier | The verifier run in a fresh sandbox that gets only the workspace and artifacts. | [Verifiers](https://www.benchflow.ai/docs/benchflow/tasks/verifiers.md#run-the-verifier-in-its-own-sandbox) |
| Skill | Instructions and files an agent can load for a kind of task. | [Skills](https://www.benchflow.ai/docs/benchflow/evaluate/skills.md) |
| Soft check | The task's test run between rounds or steps, on the workspace as the agent left it; the final check is the one that counts. | [Multi-turn tasks](https://www.benchflow.ai/docs/benchflow/tasks/multi-turn.md#turns-and-rounds) |
| Task | A folder with an instruction, an environment, a verifier and usually a reference solution. | [Task format](https://www.benchflow.ai/docs/benchflow/tasks/task-format.md) |
| Trajectory | Everything the agent did in a trial: messages, thoughts, tool calls. | [Read the results](https://www.benchflow.ai/docs/benchflow/evaluate/read-results.md) |
| Trial | The command line's and the result folders' name for a rollout. The `--trials` flag and the `trial-NN` folders count repeats instead (see Repeat). | [How BenchFlow works](https://www.benchflow.ai/docs/benchflow/start/how-it-works.md) |
| Turn | One prompt to one agent. | [Multi-turn tasks](https://www.benchflow.ai/docs/benchflow/tasks/multi-turn.md#turns-and-rounds) |
| Unscored | A trial whose agent finished but whose verifier couldn't score it. | [When a trial fails](https://www.benchflow.ai/docs/benchflow/evaluate/failed-trials.md) |
| Verifier | The task's check, usually `verifier/test.sh`, which writes the reward. | [Verifiers](https://www.benchflow.ai/docs/benchflow/tasks/verifiers.md) |
| Warm pool | Sandboxes prepared ahead of their trial's turn on Docker: 1 by default, 0 with `-j 1`; `--warm-pool` sets the number. | [Concurrency and budgets](https://www.benchflow.ai/docs/benchflow/scale/cost.md#concurrency) |
| Workspace | The folder the agent works in (the task's `WORKDIR`), which the verifier grades. | [How BenchFlow works](https://www.benchflow.ai/docs/benchflow/start/how-it-works.md) |