Task format

A task is a folder: an instruction, an environment for the sandbox, a verifier that scores the result, and usually a reference solution. Here are its files and what each is for; where a task has two forms of a part, they must match. Every setting with its type and default is in Files BenchFlow reads and writes.

The files

The half-life task has every part a task usually needs:

half-life/
├── task.md                 settings on top, the instruction below
├── environment/
│   ├── Dockerfile          builds the sandbox
│   └── data/counts.csv     files the Dockerfile copies in
├── verifier/
│   ├── test.sh             scores the trial: writes /logs/verifier/reward.txt
│   └── rubric.json         optional: a rubric a reviewer agent grades against
└── oracle/
    └── solve.sh            the reference solution the oracle agent runs

The task.md header

The header is YAML between two --- lines:

schema_version: "1.3"
metadata:
  author_name: BenchFlow docs
  difficulty: medium
  category: capability
  tags: [physics, data-analysis]
agent:
  timeout_sec: 900       # the agent's time limit, in seconds
verifier:
  timeout_sec: 120       # the verifier's time limit
sandbox:
  cpus: 1
  memory_mb: 2048

Below the header is the instruction. The body can be split at reserved headings: ## prompt holds the instruction, and ## role:<name>, ## scene:<name> and ## user-persona hold the parts of a multi-turn task. A body with none of them is the instruction as a whole.

The header's top-level keys are schema_version, task, metadata, agent, verifier, sandbox, oracle, source, artifacts and reward, and for multi-turn tasks agents, scenes and user. An unknown key is refused, so a typo can't silently change a run. steps describes a multi-step task, which only 0.10.0rc2 runs, and 0.10.0rc2 adds preference for human preferences.

The environment

With an environment/Dockerfile, BenchFlow builds the sandbox from it, with environment/ as the build context. A task that needs no build can name an image instead:

sandbox:
  docker_image: python:3.12-slim
  workdir: /app

The agent works as a user named agent, not root, in the image's working directory: its WORKDIR, or /root when it sets none. Before the agent starts, BenchFlow gives that folder and everything in it to the agent user; files elsewhere in the image stay root's. So keep data the agent must not change outside the working directory, and install in the Dockerfile what the agent would otherwise need root for. (Checked on 0.9.1 and 0.10.0rc1, with a built image and a named one.)

Services the agent works with

A task whose agent needs other services, such as a database or a mock API, puts a docker-compose.yaml (or compose.yaml) next to the Dockerfile. Docker, remote Docker and Daytona run its services; Modal, AgentCore and Apple Container run one container, so they refuse such a task before it starts rather than run it without its services.

Other task formats

BenchFlow also reads the task.toml layout that Terminal-Bench and many other benchmarks use. The files map one to one:

When both forms of a part exist in one task, they must match: bench check and bench run refuse a task whose verifier/ and tests/ differ, or whose task.md disagrees with its task.toml. bench import converts a task.toml task to task.md; Bring what you have covers running such benchmarks as they are.