Agents and models

An agent is the program that does the task: Claude Code, Codex, OpenCode and dozens more. You choose it with -a, its model with -m, and how hard the model thinks with -e. Each agent needs a login or an API key for its model provider.

Pick an agent

These are the agents people run most, by downloads and by trials on BenchFlow:

Two more need no model: oracle runs the task's reference solution and nop does nothing. bench agents lists every agent, including GitHub Copilot CLI and others that come from the agents repository, and bench agents <name> shows one:

bench agents claude

claude-agent-acp
  Aliases:     claude
  Description: Claude Code via ACP (Anthropic's Agent Client Protocol)
  Protocol:    acp
  Launch:      /opt/benchflow/bin/claude-agent-acp
  Requires:    ANTHROPIC_API_KEY (or login)
  Provider auth: Provider-prefixed models may use different credentials; Azure Foundry models use AZURE_API_KEY + 
AZURE_API_ENDPOINT.
  Auth:        subscription via ~/.claude/.credentials.json

BenchFlow installs the agent inside the sandbox; you don't need it on your machine. It talks to most agents over the Agent Client Protocol (ACP), which is why many names end in -acp.

Log in or set a key

The subscription commands are the agents' own: claude setup-token and claude need Claude Code installed on your machine, and codex login needs the Codex CLI. An API key needs neither. bench doctor reports which logins it found (Install shows its output). A subscription login only works with models its plan includes.

Choose the model and effort

bench run <task> -a claude -m sonnet

-m takes the model in the agent's own naming. In 0.9.2, Claude Code on a subscription login offers default, sonnet, opus, haiku and claude-fable-5-1; any other name stops the trial before a model call, with agent integration failure [agent_model] and the names it offers (Quick start). -e (or --effort) sets reasoning effort for agents that have one: low, medium, high, max and so on, as the agent names them. How it reaches the agent in 0.9.2, from its agent list:

An agent with no such option stops the trial instead: reasoning_effort='max' was requested for agent '...', but that agent does not declare or advertise an ACP effort config option. The values each option takes are the agent's own.

Pass -m. Without it, an agent falls back to a default model, which for Claude Code in 0.9.2 is claude-haiku-4-5-20251001, an older one (0.10.0rc2's is claude-haiku-5-5).

The model and effort you pass are recorded in the trial's config.json (model, reasoning_effort). What the provider actually received is in trajectory/llm_trajectory.jsonl, one line per model call. Check it when a score depends on the setting: a provider can ignore one it doesn't support.

A model from another provider

Name the provider as a prefix, and export its key. OpenCode on DeepSeek:

export DEEPSEEK_API_KEY=...
bench run skillsbench/tasks/econ-detrending-correlation -a opencode -m deepseek/deepseek-v4-flash

skillsbench/ is a clone of SkillsBench at commit f860ecf0 (how).

[econ-detrending-correlation]   → end_turn, 14 total tool calls
[econ-detrending-correlation] Running verifier...
[econ-detrending-correlation] Rewards: {'reward': 1.0}
[PASS] econ-detrending-correlation (reward=1.00, tools=14)
Job complete: 1/1 (100.0%), mean_reward=1.00, errors=0, idle_timeouts=0, time=2.2min

✓ Score: 1/1 (100.0%), mean reward 1.00, errors=0
Outcomes: 1 scored (1 passed, 0 failed), 0 unscored, 0 errored
Cost:      $0.01, 247,840 tokens
Time:      2m 14s

Claude Code can use a provider that serves Anthropic's API. The Quick start runs it on Xiaomi's MiMo with -m xiaomi/mimo-v2.6-flash. Codex needs a provider that serves OpenAI's Responses API, so it can't use MiMo.

The provider prefixes

These are the prefixes 0.9.2 knows, from its own provider list. A name with no prefix, such as sonnet, goes to the agent's own service; one that starts with a family in the last column, such as deepseek-v4-flash, goes to that provider without one. Claude Code and Codex are refused before the run starts when the provider doesn't serve their API; the other agents name no API, and BenchFlow doesn't check them. bench run tasks/half-life -a codex -m xiaomi/mimo-v2.6-flash:

Agent 'codex-acp' requires provider protocol 'openai-responses', but provider 'xiaomi' for model 'xiaomi/mimo-v2.6-flash' only supports anthropic-messages, openai-completions. Use a provider prefix that matches the agent protocol.
No job was created.

A model you serve

For a model behind your own OpenAI-compatible server (vLLM, SGLang), use the vllm/ or sglang/ prefix and pass the server's address to the agent:

export OPENAI_API_KEY=...        # whatever your server expects
bench run <task> -a opencode -m vllm/<served-model-name> \
  --agent-env BENCHFLOW_PROVIDER_BASE_URL=http://<host>:8000/v1

To show the flags with a real run, we pointed this at DeepSeek's OpenAI-compatible API in place of a server of our own:

OPENAI_API_KEY=$DEEPSEEK_API_KEY bench run skillsbench/tasks/econ-detrending-correlation \
  -a opencode -m vllm/deepseek-v4-flash --agent-env BENCHFLOW_PROVIDER_BASE_URL=https://api.deepseek.com/v1

[PASS] econ-detrending-correlation (reward=1.00, tools=8)
✓ Score: 1/1 (100.0%), mean reward 1.00, errors=0
Cost:      $0.01, 121,186 tokens

Pin the agent's version

Each BenchFlow release installs a fixed version of each agent, so two runs on the same release run the same agent. 0.9.2 installs Claude Code 2.1.280 (through claude-agent-acp 0.81.2) and Codex through codex-acp 2.0.1, and 0.10.0rc2 installs Claude Code 2.1.316; bench doctor prints the versions under "Agent versions". To change an agent's version, change BenchFlow's version. On 0.9.2, a trial's result.json doesn't record the agent's version (0.10.0rc2's records it as harness_versions); its agent/install-stdout.txt does, in the install command it ran (claude-agent-acp@0.81.2, claude-code@2.1.280). Write the version down with your results (Before you run an eval).

Run Claude Code or Codex through their own CLI

--harness native runs Claude Code or Codex through its own command-line program in headless JSON mode, instead of through its ACP adapter. Use it to compare against a published number that used the plain CLI. The default, --harness acp, is what the rest of these docs use.

Codex Apps

Codex can connect account tools such as Drive or Gmail ("Codex Apps"). Scored Codex runs turn them off, so an agent can't reach your accounts or data outside the task. Each trial records the policy it ran with.