Stop reward hacking

An agent rewarded for a passing test will, sooner or later, try to pass it without doing the work: edit the test, plant a file the test imports, write the reward itself, or find the answer somewhere.

What BenchFlow does between the agent and the verifier

The agent works as its own user, not root, and never sees the tests: verifier/ is copied in only after the agent stops. Then, before test.sh runs, BenchFlow:

• stops every process the agent left running (kill-before-grading.json in the trial records it);
• removes start-up files the agent could plant to run code inside the test, such as conftest.py, sitecustomize.py and .pth files, and modules named after a test runner, such as pytest.py;
• deletes any reward file already in /logs/verifier/;
• gives the test a fresh, root-only home folder when the agent could write to root's;
• removes the git hooks, filters, aliases and other command settings the agent added to a repository in the workspace, and runs root's git with hooks, fsmonitor, external diff and signing off, from 0.9.2 on. A task whose answer is a hook or a filter sets cleanup_git: false under verifier: hardening:.

What it can't stop

These are the task author's to close:

• A test that trusts a file the agent can write. If the test reads its expected value, or the data it checks against, from the workspace, the agent can change it. Keep expected values in verifier/.
• A test that imports from the workspace. Python run from the workspace imports the agent's modules first; run a script with python3 -I, and pytest with $BENCHFLOW_PYTEST (Verifiers).
• Answers the agent can look up. With open network access, an agent can fetch a published solution. Network and secrets shows how to close it.
• A grader that checks only the answer. The half-life test can't tell a fitted 37.4 from a typed one; a rubric review can.

Probe your verifier

Write the cheats you're worried about as small scripts, and run them as the agent with -a script:<file>. A script agent runs the file in the sandbox as the agent's user, with no model. Three probes against the half-life task; the task and the probes come from the BenchFlow repository's examples:

git clone --depth 1 --branch v0.10.0rc3 https://github.com/benchflow-ai/benchflow.git
mkdir -p tasks && cp -r benchflow/docs/examples/research-task/half-life tasks/
# these probes test the test; a rubric review needs a reviewer model
rm tasks/half-life/verifier/rubric.json
cp -r benchflow/docs/examples/research-task/probes .
bench run tasks/half-life -a script:probes/look.sh
bench run tasks/half-life -a script:probes/shadow-json.sh
bench run tasks/half-life -a script:probes/write-reward.sh

A script agent's output is in the trial's agent/script.txt. The look.sh probe shows what an agent can find while it works:

/verifier: absent
/tests: absent
/oracle: absent
/solution: absent
/logs/verifier: 
test.sh, solve.sh or rubric.json anywhere: 
files in /app, /home and /tmp that contain 37.4:

No tests, no reference solution, no file holding the answer. The json.py probe fails because the test runs python3 -I; without -I it would pass. The reward written by the agent is deleted before the test runs. The probe scripts are in docs/examples/research-task/probes. A task holds when every probe scores 0 while the oracle scores 1.

Audit how each reward was earned

--integrity audit records what the agent did to the files and network the reward depends on, checks it against the task's rules, and writes a verdict next to each reward. The reward itself never changes.

bench run tasks/half-life -a script:probes/write-reward.sh --integrity audit

The run prints the reward as usual ([FAIL] half-life (reward=0.00, tools=0)); the verdict is in the trial's integrity/claim_verdict.json. Its main fields:

{
  "core_verdict": "VectorExposed",
  "exploited": false,
  "reason": "no violation observed; exposed by shared_verifier, access_unattested (pre-turn probe scripted)",
  "reward": 0.0,
  "reward_effect": "none",
  "reward_source": "TrustedVerifier"
}

core_verdict is one of four: Checked (nothing the agent did reached the reward, and every way it could have was closed), VectorExposed (nothing reached it, but a way stayed open), AgentViolation (something the agent did reached it) or Rejected (the evidence to judge is incomplete). Nothing the probe did reached the reward (reward_effect: "none"): its reward.txt was deleted before the test ran, and the reward came from BenchFlow's own run of the test. The verdict is VectorExposed because of how the trial ran, not what the agent did: the test shared the agent's sandbox (shared_verifier), and for a script agent no root check confirmed that the protected paths were closed to it (access_unattested). The audit judges what reached the reward, so the attempt itself isn't flagged; the trajectory shows it.

--integrity strict also runs the test in a separate sandbox that receives only the agent's outputs, on the sandboxes that can: Docker, remote Docker, Daytona and Modal (Sandboxes). The oracle's verdict on the half-life task is then:

{"core_verdict": "Checked", "reason": "no violation observed; the separate verifier received only the transferred outputs", "reward": 1.0}

To run the probes over a whole benchmark you didn't write, see Check a benchmark.