Write a task

This tutorial builds a small research task the way a physics research benchmark does: the agent gets real data, must produce a number with an uncertainty, and a test checks the answer against a tolerance. You'll write the task, prove it with its reference solution and an agent that does nothing, then run Claude Code on it. Building and checking it takes about twenty minutes; the three Claude Code tries at the end run up to 15 minutes each. You need BenchFlow installed and Docker.

Runs here use MiMo (-m xiaomi/mimo-v2.6-flash): export its key and base URL first (Quick start), or use -m sonnet with a Claude login.

The finished task is in the repository at docs/examples/research-task/half-life, with the files used to test it. Other guides run it as tasks/half-life; to follow them without building it, copy it:

git clone --depth 1 --branch v0.10.0rc3 https://github.com/benchflow-ai/benchflow.git
mkdir -p tasks && cp -r benchflow/docs/examples/research-task/half-life tasks/
rm tasks/half-life/verifier/rubric.json   # Rubric review adds it; the other guides run without it

The task

A radioactive sample was counted every minute for two hours, in a room with a constant background that wasn't measured. The agent must estimate the half-life, with a one-standard-deviation uncertainty. The data were generated with a half-life of 37.4 minutes, so the test knows the answer; the agent doesn't.

A good test for this has to fail the tempting wrong answer. A fit that ignores the background gets 41.9 ± 0.4 minutes; one that models it gets 37.1 ± 1.1. The test below fails the first and passes the second.

Scaffold it

bench new half-life

Created: tasks/half-life/
  environment/Dockerfile
  oracle/solve.sh
  task.md
  verifier/rubrics/verifier.md
  verifier/rubrics/verifier.toml
  verifier/test.sh
  verifier/test_outputs.py
  verifier/verifier.md
Next: replace every [REPLACE: ...] placeholder (bench check lists them), then prove the task with its reference solution:
bench check tasks/half-life
bench run tasks/half-life --sandbox docker --jobs-dir jobs/half-life-oracle --fresh

Every file starts with a placeholder that fails on purpose, so an unfinished task can't pass. task.md holds the settings and the instruction, environment/Dockerfile the sandbox and oracle/solve.sh the reference solution. The verifier folder holds more than one task needs: test.sh is the test, which writes the reward; test_outputs.py is a pytest file, though the scaffold's test.sh never runs it (to keep it, call it from test.sh as Verifiers shows; bench new --no-pytest leaves it out); and verifier.md with rubrics/ is a verifier document, which combines several checks into one weighted reward. This task grades with test.sh alone, so delete the rest:

rm -r tasks/half-life/verifier/rubrics tasks/half-life/verifier/test_outputs.py tasks/half-life/verifier/verifier.md

Build the environment and add the data

The sandbox is built from environment/Dockerfile. Copy the data in and install what a solution needs:

FROM ubuntu:24.04
RUN apt-get update -qq && apt-get install -y -qq python3 python3-numpy && rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY data/ /app/data/
RUN mkdir -p /logs/verifier /logs/agent /logs/artifacts

The data are 120 rows of minute,counts, generated with a half-life of 37.4 minutes. Download them into the task:

mkdir -p tasks/half-life/environment/data
curl -fsSL -o tasks/half-life/environment/data/counts.csv \
  https://raw.githubusercontent.com/benchflow-ai/benchflow/v0.10.0rc3/docs/examples/research-task/half-life/environment/data/counts.csv

Everything in the image is visible to the agent, so nothing that gives away the answer goes here.

Write the instruction

task.md holds the settings between the --- lines and the instruction below them:

---
schema_version: "1.3"
metadata:
  author_name: BenchFlow docs
  difficulty: medium
  category: capability
  tags: [physics, data-analysis]
agent:
  timeout_sec: 900
verifier:
  timeout_sec: 120
sandbox:
  cpus: 1
  memory_mb: 2048
---
# half-life

## prompt

The file `/app/data/counts.csv` holds the counts recorded in each minute of a two-hour measurement of a radioactive sample (columns `minute` and `counts`). The room adds a constant background, which was not measured separately.

Estimate the sample's half-life in minutes, with a one-standard-deviation uncertainty. Write `/app/answer.json` as `{"half_life_min": <number>, "uncertainty_min": <number>}`. Write `/app/report.md`: how you estimated both numbers, how you handled the background and the counting noise, and how to reproduce your result. Leave the script you used in `/app`.

agent.timeout_sec is the agent's time limit, 15 minutes here. Task format lists every setting.

The instruction states the standard to meet, an estimate with an honest uncertainty, and not the grader's numbers: it doesn't give the 2.5-minute tolerance or the accepted range of uncertainties, which come from running a right and a wrong method against the test (below). If your benchmark wants agents to see them, put them in the instruction; either way, write the choice down.

Write the test

verifier/test.sh runs after the agent stops. It reads the answer and writes the reward, 1 or 0, to /logs/verifier/reward.txt:

#!/bin/bash
# -I: import nothing from the working directory, which is the agent's workspace (/app).
python3 -I - <<'PY'
import json

TRUE = 37.4  # minutes: the half-life the data were generated with
reward = 0
try:
    answer = json.load(open("/app/answer.json"))
    est, unc = float(answer["half_life_min"]), float(answer["uncertainty_min"])
    off = abs(est - TRUE)
    # A fit that models the background gives 37.1 +- 1.1; one that ignores it gives 41.9 +- 0.4.
    checks = [
        (off <= 2.5, f"{off:.2f} min from the true value, more than 2.5"),
        (0.3 <= unc <= 3.0, f"uncertainty {unc:.2f} min, outside 0.3 to 3"),
        (unc > 0 and off <= 3 * unc,
         f"the true value is {off / unc:.1f} standard deviations away, more than 3" if unc > 0
         else "the uncertainty is not positive"),
    ]
    failed = [why for ok, why in checks if not ok]
    reward = 0 if failed else 1
    verdict = "FAIL: " + "; ".join(failed) if failed else "PASS"
    print(f"{verdict} (answer {est:.2f} +- {unc:.2f} min, true value {TRUE})")
except (OSError, ValueError, KeyError, TypeError) as exc:
    print(f"FAIL: no usable /app/answer.json ({exc})")
open("/logs/verifier/reward.txt", "w").write(f"{reward}\n")
PY

Three choices in it matter:

• The tolerances are stated and calibrated. 2.5 minutes passes the right method's 37.1 and fails the background-free 41.9. The uncertainty must be plausible (0.3 to 3 minutes) and must cover the true value within three standard deviations, so a wide guess and an overconfident one both fail.
• It prints the verdict first. test-stdout.txt then says why a trial failed in one line.
• python3 -I. The test runs in the agent's workspace. Without -I, a json.py the agent left in /app would replace Python's own json module and could answer for the test. Stop reward hacking shows this attack failing against this test. A test that runs pytest uses $BENCHFLOW_PYTEST instead (Verifiers).

Write the reference solution

oracle/solve.sh solves the task the way an expert would; the oracle agent runs it. This one writes and runs a weighted least-squares fit of A·exp(−ln2·t/T) + B, with Poisson weights, takes the uncertainty from the chi-square profile, and writes the answer and a report:

#!/bin/bash
set -e
cat > /app/fit.py <<'PY'
"""Weighted least squares of counts = A exp(-ln2 t / T) + B, profiled over T."""
import json

import numpy as np

d = np.loadtxt("/app/data/counts.csv", delimiter=",", skiprows=1)
t, c = d[:, 0] + 0.5, d[:, 1]  # each minute's centre
w = 1 / np.maximum(c, 1)  # Poisson: the variance of a count is the count


def chi2(T):
    X = np.column_stack([np.exp(-np.log(2) * t / T), np.ones_like(t)])
    coef, *_ = np.linalg.lstsq(X * np.sqrt(w)[:, None], c * np.sqrt(w), rcond=None)
    return float(np.sum(w * (c - X @ coef) ** 2))


grid = np.arange(20, 80, 0.005)
vals = np.array([chi2(T) for T in grid])
best = float(grid[vals.argmin()])
inside = grid[vals <= vals.min() + 1]  # 1 sigma: chi-square within 1 of its minimum
unc = float(inside.max() - inside.min()) / 2
json.dump({"half_life_min": round(best, 3), "uncertainty_min": round(unc, 3)}, open("/app/answer.json", "w"))
print(f"half-life {best:.3f} +- {unc:.3f} min")
PY
python3 /app/fit.py
cat > /app/report.md <<'MD'
# Half-life

- Model: counts in each minute = A exp(-ln2 t / T) + B, with B the constant background, at each minute's centre.
- Fit: weighted least squares with Poisson weights (variance = counts); A and B are solved for each T, and T is scanned from 20 to 80 minutes.
- Uncertainty: half the width of the range of T where chi-square is within 1 of its minimum.
- Reproduce: `python3 /app/fit.py`.
MD

Check it and run the controls

bench check reads the task without starting a sandbox:

bench check tasks/half-life

✓ half-life — valid (structural)

valid (structural) means the files and settings are well formed; it doesn't say the test is right. The controls do that. Then the two controls. The reference solution must score 1, and doing nothing must score 0:

bench run tasks/half-life -a oracle
bench run tasks/half-life -a nop --fresh

✓ Score: 1/1 (100.0%), mean reward 1.00, errors=0
...
• Score: 0/1 (0.0%), mean reward 0.00, errors=0
nop control: 0/1, as expected: doing nothing passes none of the tasks

A task that fails either control can't tell you anything about an agent. The test also has to fail wrong answers; we ran three wrong solutions against it as scripted agents, and it failed each one:

A fourth one passes: the right number with no method behind it (PASS (answer 37.40 +- 1.00 min, true value 37.4)). A test of the number can't see how it was found. Rubric review adds a check of the method to this task. Check a benchmark shows how to run wrong solutions like these as agents.

Run an agent on it

Three tries of Claude Code on MiMo V2.6 Flash, with each reward's integrity audited (Stop reward hacking):

bench run tasks/half-life -a claude -m xiaomi/mimo-v2.6-flash --trials 3 --integrity audit

[half-life] Agent prompt exceeded wall-clock budget 900s
[half-life] Running verifier...
[half-life] Rewards: {'reward': 1.0}
[PASS] half-life (reward=1.00, tools=16) (Agent prompt exceeded wall-clock budget 900s)
...
Matrix: 3 trial job(s), 3/3 trials passed, errors=0 verifier-errors=0

All three passed, and all three ran into the 15-minute limit: the agent wrote its answer early and went on checking it until time ran out. The test scores only the files the agent left, so a trial that hits the limit after writing a right answer still passes. Each such trial shows timed_out in the Exec column of bench results, and its result.json records the timeout. If finishing in time matters to you, count those trials separately, or make a run fail on them with --fail-on timeout.

The audit found nothing the agent did that reached the reward in any try. Stop reward hacking shows how to read its verdicts.

Next

• Grade the report and the method as well as the number: Rubric review.
• Prove an agent can't cheat the test: Stop reward hacking.
• Share a folder of tasks as a benchmark: Publish a benchmark.