# CLI and Python

> Find the Python argument that does what each `bench` flag does.

Docs index for agents: https://www.benchflow.ai/docs/llms.txt

Each flag of `bench eval run`, `bench eval branch`, `bench eval metrics`, `bench eval compare`, `bench eval regrade`, `bench train convert`, `bench train stream`, and the Python that does the same. `sdk` is a name in the public `benchflow` package, `module` a function or field in one of its modules, and `cli` a flag with no Python equivalent, for the reason given. `bench run` takes the flags of `bench eval run`, some under shorter names: [Commands](https://www.benchflow.ai/docs/benchflow/reference/commands.md) has both, and [Python API](https://www.benchflow.ai/docs/benchflow/reference/python.md) every public name. A command or flag new since 0.9.2 has `(0.10)` after it when 0.10.0rc3 has it, or `(unreleased)` when no release has it yet.


## `bench eval run`

| Flag | Kind | Python | Note |
|---|---|---|---|
| `--config` | sdk | `benchflow.Evaluation.from_yaml` | also `Evaluation.from_dict()`; `to_yaml()/to_dict()` write it |
| `--tasks-dir` | sdk | `benchflow.Evaluation.__init__(tasks_dir)` |  |
| `--source-repo` | sdk | `benchflow.resolve_source(repo)` | pass the returned path as tasks_dir |
| `--source-path` | sdk | `benchflow.resolve_source(path)` |  |
| `--source-ref` | sdk | `benchflow.resolve_source(ref)` |  |
| `--source-env` | module | `benchflow.hosted_env.HostedEnvRunConfig.source_env` | run with `benchflow.hosted_env.run_hosted_env()` |
| `--source-env-version` | module | `benchflow.hosted_env.HostedEnvRef.version` |  |
| `--source-env-arg` | module | `benchflow.hosted_env.HostedEnvRunConfig.env_args` |  |
| `--source-env-num-examples` | module | `benchflow.hosted_env.HostedEnvRunConfig.num_examples` |  |
| `--source-env-rollouts-per-example` | module | `benchflow.hosted_env.HostedEnvRunConfig.rollouts_per_example` |  |
| `--source-env-max-tokens` | module | `benchflow.hosted_env.HostedEnvRunConfig.max_tokens` |  |
| `--source-env-temperature` | module | `benchflow.hosted_env.HostedEnvRunConfig.temperature` |  |
| `--source-env-sampling-arg` | module | `benchflow.hosted_env.HostedEnvRunConfig.sampling_args` |  |
| `--source-env-verifiers-version` | module | `benchflow.hosted_env.HostedEnvRunConfig.verifiers_version` |  |
| `--source-env-base-url` | module | `benchflow.hosted_env.HostedEnvRunConfig.api_base_url` |  |
| `--source-env-api-key-var` | module | `benchflow.hosted_env.HostedEnvRunConfig.api_key_var` |  |
| `--agent` | sdk | `benchflow.EvaluationConfig.agent` |  |
| `--model` | sdk | `benchflow.EvaluationConfig.model` |  |
| `--reasoning-effort` | sdk | `benchflow.EvaluationConfig.reasoning_effort` |  |
| `--harness` | sdk | `benchflow.EvaluationConfig.harness` | acp (default) or native |
| `--sandbox` | sdk | `benchflow.EvaluationConfig.environment` |  |
| `--usage-tracking` | sdk | `benchflow.EvaluationConfig.usage_tracking` |  |
| `--environment-manifest` | sdk | `benchflow.EvaluationConfig.environment_manifest` | bf.load_manifest(path or name@version) |
| `--state` | module | `benchflow._utils.env_registry.resolve_state(value)` | its manifest goes in EvaluationConfig.environment_manifest |
| `--prompt` | sdk | `benchflow.EvaluationConfig.prompts` |  |
| `--config-override` | sdk | `benchflow.EvaluationConfig.config_override` |  |
| `--concurrency` | sdk | `benchflow.EvaluationConfig.concurrency` |  |
| `--build-concurrency` | sdk | `benchflow.EvaluationConfig.build_concurrency` |  |
| `--warm-pool` | sdk | `benchflow.EvaluationConfig.warm_pool` |  |
| `--worker-concurrency` | module | `benchflow.eval_sharding.run_sharded_evaluation(worker_concurrency)` |  |
| `--worker-retries` | module | `benchflow.eval_sharding.run_sharded_evaluation(worker_retries)` |  |
| `--worker-start-stagger-sec` | module | `benchflow.eval_sharding.run_sharded_evaluation(worker_start_stagger_sec)` |  |
| `--agent-idle-timeout` | sdk | `benchflow.EvaluationConfig.agent_idle_timeout` |  |
| `--checkpoints` | sdk | `benchflow.EvaluationConfig.checkpoints` |  |
| `--checkpoint-keep` | sdk | `benchflow.EvaluationConfig.checkpoint_keep` |  |
| `--freeze-workspace` | sdk | `benchflow.EvaluationConfig.freeze_workspace` |  |
| `--keep-sandbox` (0.10) | sdk | `benchflow.EvaluationConfig.keep_sandbox` |  |
| `--accept-deviations` (0.10) | sdk | `benchflow.EvaluationConfig.accept_deviations` |  |
| `--fresh-install` | sdk | `benchflow.EvaluationConfig.fresh_install` |  |
| `--allow-cloud-metadata` | sdk | `benchflow.EvaluationConfig.allow_cloud_metadata` |  |
| `--integrity` | sdk | `benchflow.EvaluationConfig.integrity` |  |
| `--verifier-sandbox` | sdk | `benchflow.EvaluationConfig.verifier_sandbox` |  |
| `--quiet` | cli |  | console output only; Python reports through the logging module |
| `--jobs-dir` | sdk | `benchflow.Evaluation.__init__(jobs_dir)` |  |
| `--fresh` | sdk | `benchflow.Evaluation.__init__(job_name)` | pass a new job_name |
| `--job-name` | sdk | `benchflow.Evaluation.__init__(job_name)` |  |
| `--restart-changed` (0.10) | sdk | `benchflow.Evaluation.__init__(restart_changed)` | also `Evaluation.resume(restart_changed=)` |
| `--fail-under` | sdk | `benchflow.EvaluationResult.score_of_plan` | compare it to your threshold |
| `--fail-on` | sdk | `benchflow.EvaluationResult.results` | count results by error_category / errored / verifier_errored |
| `--summary-out` | module | `benchflow.job_export.run_summary_export(result)` | the benchflow.run-summary document of an EvaluationResult |
| `--codex-apps-policy` | sdk | `benchflow.EvaluationConfig.codex_apps_policy` |  |
| `--sandbox-user` | sdk | `benchflow.EvaluationConfig.sandbox_user` |  |
| `--sandbox-setup-timeout` | sdk | `benchflow.EvaluationConfig.sandbox_setup_timeout` |  |
| `--context-root` | sdk | `benchflow.EvaluationConfig.context_root` |  |
| `--base-image-override` | sdk | `benchflow.EvaluationConfig.base_image_override` |  |
| `--separate-verifier` | sdk | `benchflow.EvaluationConfig.separate_verifier` |  |
| `--separate-verifier-include` | sdk | `benchflow.EvaluationConfig.separate_verifier_include` |  |
| `--skills-dir` | sdk | `benchflow.EvaluationConfig.skills_dir` |  |
| `--skill-mode` | sdk | `benchflow.EvaluationConfig.skill_mode` |  |
| `--skill-creator-dir` | sdk | `benchflow.EvaluationConfig.skill_creator_dir` |  |
| `--self-gen-no-internet` | sdk | `benchflow.EvaluationConfig.self_gen_no_internet` |  |
| `--loop-strategy` | sdk | `benchflow.EvaluationConfig.loop_strategy` |  |
| `--reviewer-agent` | sdk | `benchflow.ReviewerConfig.agent` | EvaluationConfig.reviewer |
| `--reviewer-model` | sdk | `benchflow.ReviewerConfig.model` |  |
| `--reviewer-reasoning-effort` | sdk | `benchflow.ReviewerConfig.reasoning_effort` |  |
| `--reviewer-sandbox` | sdk | `benchflow.ReviewerConfig.environment` |  |
| `--reviewer-timeout-sec` | sdk | `benchflow.ReviewerConfig.timeout_sec` |  |
| `--reviewer-idle-timeout` | sdk | `benchflow.ReviewerConfig.idle_timeout_sec` |  |
| `--reviewer-storage-mb` (0.10) | sdk | `benchflow.ReviewerConfig.storage_mb` |  |
| `--reviewer-memory-mb` (0.10) | sdk | `benchflow.ReviewerConfig.memory_mb` |  |
| `--reviewer-setup-timeout` (0.10) | sdk | `benchflow.ReviewerConfig.setup_timeout_sec` |  |
| `--reviewer-evidence-timeout` (0.10) | sdk | `benchflow.ReviewerConfig.evidence_timeout_sec` |  |
| `--reviewer-concurrency` | sdk | `benchflow.ReviewerConfig.concurrency` |  |
| `--reviewer-image` | sdk | `benchflow.ReviewerConfig.image` |  |
| `--reviewer-agent-env` | sdk | `benchflow.ReviewerConfig.agent_env` |  |
| `--reviewer-open-network` | sdk | `benchflow.ReviewerConfig.open_network` |  |
| `--agent-env` | sdk | `benchflow.EvaluationConfig.agent_env` |  |
| `--allow-task-env` | module | `benchflow.task.env.allow_task_env(names)` | or BENCHFLOW_ALLOW_TASK_ENV=NAME,NAME in the environment |
| `--include` | sdk | `benchflow.EvaluationConfig.include_tasks` |  |
| `--exclude` | sdk | `benchflow.EvaluationConfig.exclude_tasks` |  |
| `--dataset` | module | `benchflow._utils.dataset_registry.resolve_dataset(spec)` | pass its tasks dir as tasks_dir and its name/version/digests as EvaluationConfig.dataset_* |
| `--registry` | module | `benchflow._utils.dataset_registry.resolve_dataset(registry)` |  |
| `--ignore-bench-version` | cli |  | the CLI's bench-version gate for --dataset; `resolve_dataset()` does not check the bench version |
| `--task-manifest-out` | module | `benchflow.eval_artifacts.write_task_manifest(path)` |  |
| `--run-config-out` | sdk | `benchflow.Evaluation.to_dict` | the CLI's run-config JSON also records the retry flags |
| `--health-summary-out` | module | `benchflow.eval_artifacts.write_health_summary(job_dir)` |  |
| `--expected-tasks` | module | `benchflow.eval_artifacts.write_canonical_selection(expected_tasks)` |  |
| `--canonicalize` | module | `benchflow.eval_artifacts.write_canonical_selection(policy)` |  |
| `--canonical-selection-out` | module | `benchflow.eval_artifacts.write_canonical_selection(path)` |  |
| `--canonical-jobs-dir` | module | `benchflow.eval_artifacts.materialize_canonical_job(output_dir)` |  |
| `--retry-from-checkpoint` | sdk | `benchflow.EvaluationConfig.retry_from_checkpoint` | needs checkpoints |
| `--retry-prompt` | sdk | `benchflow.EvaluationConfig.retry_prompt` |  |
| `--retry-resume-session` | sdk | `benchflow.EvaluationConfig.retry_resume_session` |  |
| `--max-cost-usd` | sdk | `benchflow.Budget.max_cost_usd` | `Evaluation(budget=Budget(...))` |
| `--max-sandbox-seconds` | sdk | `benchflow.Budget.max_sandbox_seconds` |  |
| `--max-tokens` | sdk | `benchflow.Budget.max_tokens` |  |
| `--max-rollouts` | sdk | `benchflow.Budget.max_rollouts` |  |
| `--token-budget` (0.10) | sdk | `benchflow.TrialBudget.tokens` | `EvaluationConfig(trial_budget=TrialBudget(...))`; a task's agent.token_budget |
| `--turn-budget` (0.10) | sdk | `benchflow.TrialBudget.turns` | as --token-budget; a task's agent.turn_budget |
| `--retry-policy` | cli |  | reserved: only recorded in the --run-config-out file |
| `--retry-attempts` | sdk | `benchflow.RetryConfig.max_retries` | EvaluationConfig.retry |
| `--retry-concurrency` | cli |  | reserved: only recorded in the --run-config-out file |
| `--publish-hf` | module | `benchflow.publish.huggingface.publish_folder_to_hf(repo_id)` |  |
| `--hf-prefix` | module | `benchflow.publish.huggingface.publish_folder_to_hf(path_in_repo)` |  |
| `--hf-public-read-check` | module | `benchflow.publish.huggingface.publish_folder_to_hf(public_read_check)` |  |
| `--publish-bucket` | module | `benchflow.publish.huggingface.publish_folder_to_bucket(bucket_id)` |  |
| `--eval-results-model` | module | `benchflow.publish.huggingface.open_eval_results_pr(model_repo)` |  |
| `--eval-results-dataset` | module | `benchflow.publish.huggingface.open_eval_results_pr(dataset_id)` |  |
| `--eval-results-task` | module | `benchflow.publish.huggingface.open_eval_results_pr(task_id)` |  |
| `--matrix` | sdk | `benchflow.Evaluation.__init__(config)` | one Evaluation per model entry, each with its own jobs_dir |
| `--trials` | sdk | `benchflow.Evaluation.__init__(job_name)` | run the Evaluation once per trial with a distinct job_name |
| `--n-tasks` | sdk | `benchflow.EvaluationConfig.n_tasks` |  |
| `--sample-seed` | sdk | `benchflow.EvaluationConfig.sample_seed` |  |
| `--timeout-multiplier` | sdk | `benchflow.EvaluationConfig.timeout_multiplier` |  |
| `--extra-instruction` | sdk | `benchflow.EvaluationConfig.extra_instruction` |  |
| `--dry-run` | cli |  | prints the resolved plan and task selection; nothing runs |
| `--seeds` | sdk | `benchflow.EvaluationConfig.seeds` | a list of ints; benchflow.embodied.rollouts.parse_seeds reads the CLI form |

## `bench eval branch`

| Flag | Kind | Python | Note |
|---|---|---|---|
| `--tasks-dir` | sdk | `benchflow.branch(task_path)` | bf.branch runs one task per call |
| `--agent` | sdk | `benchflow.branch(agent)` |  |
| `--model` | sdk | `benchflow.branch(model)` |  |
| `--reasoning-effort` | sdk | `benchflow.branch(reasoning_effort)` |  |
| `--harness` | sdk | `benchflow.branch(harness)` |  |
| `--sandbox` | sdk | `benchflow.branch(sandbox)` |  |
| `--prompt` | sdk | `benchflow.branch(prompts)` |  |
| `--checkpoint-after-prompt` | sdk | `benchflow.branch(checkpoint_after)` |  |
| `--child` | sdk | `benchflow.branch(children)` | a label -> prompt mapping or ChildSpec items |
| `--snapshot-layers` | sdk | `benchflow.branch(snapshot_layers)` |  |
| `--parent` | sdk | `benchflow.branch(parent)` |  |
| `--retain-snapshots` | sdk | `benchflow.branch(retain_snapshots)` |  |
| `--from-checkpoint` | sdk | `benchflow.branch(from_checkpoint)` |  |
| `--fork` | sdk | `benchflow.branch(fork)` |  |
| `--checkpoints` | sdk | `benchflow.branch(checkpoints)` | a CheckpointPolicy: `benchflow.checkpoints.parse_checkpoint_policy()` |
| `--checkpoint-keep` | module | `benchflow.checkpoints.parse_checkpoint_policy(keep)` |  |
| `--resume-session` | sdk | `benchflow.branch(resume_session)` |  |
| `--child-retries` | sdk | `benchflow.branch(child_retries)` |  |
| `--stop-on-child-failure` | sdk | `benchflow.branch(continue_after_child_failure)` |  |
| `--concurrency` | sdk | `benchflow.branch(concurrency)` |  |
| `--isolate-children` | sdk | `benchflow.branch(isolate_children)` |  |
| `--agent-env` | sdk | `benchflow.branch(agent_env)` |  |
| `--include` | sdk | `benchflow.branch(task_path)` | call bf.branch once per selected task |
| `--jobs-dir` | sdk | `benchflow.branch(jobs_dir)` |  |
| `--job-name` | sdk | `benchflow.branch(job_name)` |  |

## `bench eval metrics`

| Flag | Kind | Python | Note |
|---|---|---|---|
| `--benchmark` | module | `benchflow.metrics.collect_metrics(benchmark)` |  |
| `--agent` | module | `benchflow.metrics.collect_metrics(agent)` |  |
| `--model` | module | `benchflow.metrics.collect_metrics(model)` |  |
| `--json` | sdk | `benchflow.Job.solve_rates` | `collect_metrics(...).summary()` plus `bf.load_job(dir).solve_rates().to_dict()` |
| `--k` | sdk | `benchflow.Job.solve_rates(ks)` |  |
| `--solve-threshold` | sdk | `benchflow.Job.solve_rates(solve_threshold)` |  |
| `--include-controls` | sdk | `benchflow.Job.solve_rates(include_controls)` |  |

## `bench eval compare`

| Flag | Kind | Python | Note |
|---|---|---|---|
| `--json` | sdk | `benchflow.Comparison.to_json` |  |
| `--labels` | sdk | `benchflow.compare(labels)` |  |
| `--vary` | sdk | `benchflow.compare(vary)` |  |
| `--on-mismatch` | sdk | `benchflow.compare(on_mismatch)` |  |
| `--include-controls` | sdk | `benchflow.compare(include_controls)` |  |
| `--attempts` | sdk | `benchflow.load_job(attempts)` | load each side, then compare |
| `--out` | sdk | `benchflow.Comparison.to_json(path)` |  |
| `--by` | sdk | `benchflow.compare(by)` | comma-separated on the CLI, a tuple in Python |
| `--k` | sdk | `benchflow.compare(ks)` |  |
| `--solve-threshold` | sdk | `benchflow.compare(solve_threshold)` |  |

## `bench eval regrade`

| Flag | Kind | Python | Note |
|---|---|---|---|
| `--tasks-dir` | sdk | `benchflow.regrade(tasks_dir)` |  |
| `--sandbox` | sdk | `benchflow.regrade(sandbox)` |  |
| `--concurrency` | sdk | `benchflow.regrade(concurrency)` |  |
| `--reason` | sdk | `benchflow.regrade(reason)` |  |
| `--json` | sdk | `benchflow.RegradeSummary.to_dict` | also written to regrade-summary.json |
| `--review` (0.10) | sdk | `benchflow.regrade(review)` |  |
| `--reviewer-agent` (0.10) | sdk | `benchflow.ReviewerConfig.agent` | `benchflow.regrade(reviewer)`; set fields replace the trial's recorded ones |
| `--reviewer-model` (0.10) | sdk | `benchflow.ReviewerConfig.model` |  |
| `--reviewer-reasoning-effort` (0.10) | sdk | `benchflow.ReviewerConfig.reasoning_effort` |  |
| `--reviewer-sandbox` (0.10) | sdk | `benchflow.ReviewerConfig.environment` |  |
| `--reviewer-timeout-sec` (0.10) | sdk | `benchflow.ReviewerConfig.timeout_sec` |  |
| `--reviewer-idle-timeout` (0.10) | sdk | `benchflow.ReviewerConfig.idle_timeout_sec` |  |
| `--reviewer-storage-mb` (0.10) | sdk | `benchflow.ReviewerConfig.storage_mb` |  |
| `--reviewer-memory-mb` (0.10) | sdk | `benchflow.ReviewerConfig.memory_mb` |  |
| `--reviewer-setup-timeout` (0.10) | sdk | `benchflow.ReviewerConfig.setup_timeout_sec` |  |
| `--reviewer-evidence-timeout` (0.10) | sdk | `benchflow.ReviewerConfig.evidence_timeout_sec` |  |
| `--reviewer-concurrency` (0.10) | sdk | `benchflow.ReviewerConfig.concurrency` |  |
| `--reviewer-image` (0.10) | sdk | `benchflow.ReviewerConfig.image` |  |
| `--reviewer-agent-env` (0.10) | sdk | `benchflow.ReviewerConfig.agent_env` |  |
| `--reviewer-open-network` (0.10) | sdk | `benchflow.ReviewerConfig.open_network` |  |

## `bench train convert`

| Flag | Kind | Python | Note |
|---|---|---|---|
| `--out` | module | `benchflow.trajectories.export_prime_sft.export_prime_sft_jsonl(out)` | same parameter on the TRL and branch-tree exporters |
| `--format` | module | `benchflow.trajectories.export_prime_sft.export_prime_sft_jsonl` | one function per format: prime-sft benchflow.trajectories.export_prime_sft.export_prime_sft_jsonl, trl-sft benchflow.trajectories.export_trl_sft.export_trl_sft_jsonl, branch-tree benchflow.trajectories.export_branch.export_branch_jsonl |
| `--pairs` | module | `benchflow.trajectories.export_branch.export_branch_jsonl(pairs_out)` |  |
| `--pairs-any-request` | module | `benchflow.trajectories.export_branch.export_branch_jsonl(any_request_pairs)` |  |
| `--include-oracle` | module | `benchflow.trajectories.export_branch.export_branch_jsonl(include_oracle)` |  |
| `--min-reward` | module | `benchflow.trajectories.export_prime_sft.export_prime_sft_jsonl(min_reward)` |  |
| `--row-mode` | module | `benchflow.trajectories.export_prime_sft.export_prime_sft_jsonl(row_mode)` |  |
| `--manifest` | module | `benchflow.trajectories.export_prime_sft.export_prime_sft_jsonl(manifest)` |  |
| `--expected-rows` | module | `benchflow.trajectories.export_prime_sft.export_prime_sft_jsonl(expected_rows)` |  |
| `--canonical-selection` | module | `benchflow.trajectories.export_prime_sft.export_prime_sft_jsonl(canonical_selection)` |  |
| `--redact` | module | `benchflow.trajectories.export_prime_sft.export_prime_sft_jsonl(redact)` |  |
| `--context-policy` | module | `benchflow.trajectories.export_trl_sft.export_trl_sft_jsonl(context_policy)` |  |
| `--tokenizer` | module | `benchflow.trajectories.export_trl_sft.export_trl_sft_jsonl(tokenizer_id)` |  |
| `--tokenizer-revision` | module | `benchflow.trajectories.export_trl_sft.export_trl_sft_jsonl(tokenizer_revision)` |  |
| `--max-length` | module | `benchflow.trajectories.export_trl_sft.export_trl_sft_jsonl(max_length)` |  |
| `--subagent-rows` | module | `benchflow.trajectories.export_prime_sft.export_prime_sft_jsonl(subagent_rows)` |  |
| `--reward-vector` | module | `benchflow.trajectories.export_prime_sft.export_prime_sft_jsonl(reward_vector)` | same on the TRL exporter; `benchflow.trajectories.training_signal.rollout_training_signals()` reads the vectors without converting |
| `--group-advantage` | module | `benchflow.trajectories.export_prime_sft.export_prime_sft_jsonl(group_advantage)` | same on the TRL exporter |
| `--group-by` | module | `benchflow.trajectories.export_prime_sft.export_prime_sft_jsonl(group_by)` | comma-separated on the CLI, a tuple in Python |

## `bench train stream`

| Flag | Kind | Python | Note |
|---|---|---|---|
| `--format` | sdk | `benchflow.StreamedRollout.to_json` | jsonl is the only format; Python yields StreamedRollout objects |
| `--follow` | sdk | `benchflow.stream_rollouts(follow)` | --no-follow is follow=False |
| `--poll-interval` | sdk | `benchflow.stream_rollouts(poll_interval)` |  |
| `--timeout` | sdk | `benchflow.stream_rollouts(timeout)` | raises StreamTimeout (CLI exit 3) |
| `--group-size` | sdk | `benchflow.stream_rollouts(group_size)` |  |
| `--group-by` | sdk | `benchflow.stream_rollouts(group_by)` | comma-separated on the CLI, a tuple in Python |
| `--withheld` (0.10) | sdk | `benchflow.stream_rollouts(on_withheld)` | the CLI writes each withheld group's records to a file; Python gets the list |