CLI and Python — BenchFlow docs

Each flag of bench eval run, bench eval branch, bench eval metrics, bench eval compare, bench eval regrade, bench train convert, bench train stream, and the Python that does the same. sdk is a name in the public benchflow package, module a function or field in one of its modules, and cli a flag with no Python equivalent, for the reason given. bench run takes the flags of bench eval run, some under shorter names: Commands has both, and Python API every public name. A command or flag new since 0.9.2 has (0.10) after it when 0.10.0rc3 has it, or (unreleased) when no release has it yet.

`bench eval run`

Flag	Kind	Python	Note
`--config`	sdk	`benchflow.Evaluation.from_yaml`	also `Evaluation.from_dict()`; `to_yaml()/to_dict()` write it
`--tasks-dir`	sdk	`benchflow.Evaluation.__init__(tasks_dir)`
`--source-repo`	sdk	`benchflow.resolve_source(repo)`	pass the returned path as tasks_dir
`--source-path`	sdk	`benchflow.resolve_source(path)`
`--source-ref`	sdk	`benchflow.resolve_source(ref)`
`--source-env`	module	`benchflow.hosted_env.HostedEnvRunConfig.source_env`	run with `benchflow.hosted_env.run_hosted_env()`
`--source-env-version`	module	`benchflow.hosted_env.HostedEnvRef.version`
`--source-env-arg`	module	`benchflow.hosted_env.HostedEnvRunConfig.env_args`
`--source-env-num-examples`	module	`benchflow.hosted_env.HostedEnvRunConfig.num_examples`
`--source-env-rollouts-per-example`	module	`benchflow.hosted_env.HostedEnvRunConfig.rollouts_per_example`
`--source-env-max-tokens`	module	`benchflow.hosted_env.HostedEnvRunConfig.max_tokens`
`--source-env-temperature`	module	`benchflow.hosted_env.HostedEnvRunConfig.temperature`
`--source-env-sampling-arg`	module	`benchflow.hosted_env.HostedEnvRunConfig.sampling_args`
`--source-env-verifiers-version`	module	`benchflow.hosted_env.HostedEnvRunConfig.verifiers_version`
`--source-env-base-url`	module	`benchflow.hosted_env.HostedEnvRunConfig.api_base_url`
`--source-env-api-key-var`	module	`benchflow.hosted_env.HostedEnvRunConfig.api_key_var`
`--agent`	sdk	`benchflow.EvaluationConfig.agent`
`--model`	sdk	`benchflow.EvaluationConfig.model`
`--reasoning-effort`	sdk	`benchflow.EvaluationConfig.reasoning_effort`
`--harness`	sdk	`benchflow.EvaluationConfig.harness`	acp (default) or native
`--sandbox`	sdk	`benchflow.EvaluationConfig.environment`
`--usage-tracking`	sdk	`benchflow.EvaluationConfig.usage_tracking`
`--environment-manifest`	sdk	`benchflow.EvaluationConfig.environment_manifest`	bf.load_manifest(path or name@version)
`--state`	module	`benchflow._utils.env_registry.resolve_state(value)`	its manifest goes in EvaluationConfig.environment_manifest
`--prompt`	sdk	`benchflow.EvaluationConfig.prompts`
`--config-override`	sdk	`benchflow.EvaluationConfig.config_override`
`--concurrency`	sdk	`benchflow.EvaluationConfig.concurrency`
`--build-concurrency`	sdk	`benchflow.EvaluationConfig.build_concurrency`
`--warm-pool`	sdk	`benchflow.EvaluationConfig.warm_pool`
`--worker-concurrency`	module	`benchflow.eval_sharding.run_sharded_evaluation(worker_concurrency)`
`--worker-retries`	module	`benchflow.eval_sharding.run_sharded_evaluation(worker_retries)`
`--worker-start-stagger-sec`	module	`benchflow.eval_sharding.run_sharded_evaluation(worker_start_stagger_sec)`
`--agent-idle-timeout`	sdk	`benchflow.EvaluationConfig.agent_idle_timeout`
`--checkpoints`	sdk	`benchflow.EvaluationConfig.checkpoints`
`--checkpoint-keep`	sdk	`benchflow.EvaluationConfig.checkpoint_keep`
`--freeze-workspace`	sdk	`benchflow.EvaluationConfig.freeze_workspace`
`--keep-sandbox` (0.10)	sdk	`benchflow.EvaluationConfig.keep_sandbox`
`--accept-deviations` (0.10)	sdk	`benchflow.EvaluationConfig.accept_deviations`
`--fresh-install`	sdk	`benchflow.EvaluationConfig.fresh_install`
`--allow-cloud-metadata`	sdk	`benchflow.EvaluationConfig.allow_cloud_metadata`
`--integrity`	sdk	`benchflow.EvaluationConfig.integrity`
`--verifier-sandbox`	sdk	`benchflow.EvaluationConfig.verifier_sandbox`
`--quiet`	cli		console output only; Python reports through the logging module
`--jobs-dir`	sdk	`benchflow.Evaluation.__init__(jobs_dir)`
`--fresh`	sdk	`benchflow.Evaluation.__init__(job_name)`	pass a new job_name
`--job-name`	sdk	`benchflow.Evaluation.__init__(job_name)`
`--restart-changed` (0.10)	sdk	`benchflow.Evaluation.__init__(restart_changed)`	also `Evaluation.resume(restart_changed=)`
`--fail-under`	sdk	`benchflow.EvaluationResult.score_of_plan`	compare it to your threshold
`--fail-on`	sdk	`benchflow.EvaluationResult.results`	count results by error_category / errored / verifier_errored
`--summary-out`	module	`benchflow.job_export.run_summary_export(result)`	the benchflow.run-summary document of an EvaluationResult
`--codex-apps-policy`	sdk	`benchflow.EvaluationConfig.codex_apps_policy`
`--sandbox-user`	sdk	`benchflow.EvaluationConfig.sandbox_user`
`--sandbox-setup-timeout`	sdk	`benchflow.EvaluationConfig.sandbox_setup_timeout`
`--context-root`	sdk	`benchflow.EvaluationConfig.context_root`
`--base-image-override`	sdk	`benchflow.EvaluationConfig.base_image_override`
`--separate-verifier`	sdk	`benchflow.EvaluationConfig.separate_verifier`
`--separate-verifier-include`	sdk	`benchflow.EvaluationConfig.separate_verifier_include`
`--skills-dir`	sdk	`benchflow.EvaluationConfig.skills_dir`
`--skill-mode`	sdk	`benchflow.EvaluationConfig.skill_mode`
`--skill-creator-dir`	sdk	`benchflow.EvaluationConfig.skill_creator_dir`
`--self-gen-no-internet`	sdk	`benchflow.EvaluationConfig.self_gen_no_internet`
`--loop-strategy`	sdk	`benchflow.EvaluationConfig.loop_strategy`
`--reviewer-agent`	sdk	`benchflow.ReviewerConfig.agent`	EvaluationConfig.reviewer
`--reviewer-model`	sdk	`benchflow.ReviewerConfig.model`
`--reviewer-reasoning-effort`	sdk	`benchflow.ReviewerConfig.reasoning_effort`
`--reviewer-sandbox`	sdk	`benchflow.ReviewerConfig.environment`
`--reviewer-timeout-sec`	sdk	`benchflow.ReviewerConfig.timeout_sec`
`--reviewer-idle-timeout`	sdk	`benchflow.ReviewerConfig.idle_timeout_sec`
`--reviewer-storage-mb` (0.10)	sdk	`benchflow.ReviewerConfig.storage_mb`
`--reviewer-memory-mb` (0.10)	sdk	`benchflow.ReviewerConfig.memory_mb`
`--reviewer-setup-timeout` (0.10)	sdk	`benchflow.ReviewerConfig.setup_timeout_sec`
`--reviewer-evidence-timeout` (0.10)	sdk	`benchflow.ReviewerConfig.evidence_timeout_sec`
`--reviewer-concurrency`	sdk	`benchflow.ReviewerConfig.concurrency`
`--reviewer-image`	sdk	`benchflow.ReviewerConfig.image`
`--reviewer-agent-env`	sdk	`benchflow.ReviewerConfig.agent_env`
`--reviewer-open-network`	sdk	`benchflow.ReviewerConfig.open_network`
`--agent-env`	sdk	`benchflow.EvaluationConfig.agent_env`
`--allow-task-env`	module	`benchflow.task.env.allow_task_env(names)`	or BENCHFLOW_ALLOW_TASK_ENV=NAME,NAME in the environment
`--include`	sdk	`benchflow.EvaluationConfig.include_tasks`
`--exclude`	sdk	`benchflow.EvaluationConfig.exclude_tasks`
`--dataset`	module	`benchflow._utils.dataset_registry.resolve_dataset(spec)`	pass its tasks dir as tasks_dir and its name/version/digests as EvaluationConfig.dataset_*
`--registry`	module	`benchflow._utils.dataset_registry.resolve_dataset(registry)`
`--ignore-bench-version`	cli		the CLI's bench-version gate for --dataset; `resolve_dataset()` does not check the bench version
`--task-manifest-out`	module	`benchflow.eval_artifacts.write_task_manifest(path)`
`--run-config-out`	sdk	`benchflow.Evaluation.to_dict`	the CLI's run-config JSON also records the retry flags
`--health-summary-out`	module	`benchflow.eval_artifacts.write_health_summary(job_dir)`
`--expected-tasks`	module	`benchflow.eval_artifacts.write_canonical_selection(expected_tasks)`
`--canonicalize`	module	`benchflow.eval_artifacts.write_canonical_selection(policy)`
`--canonical-selection-out`	module	`benchflow.eval_artifacts.write_canonical_selection(path)`
`--canonical-jobs-dir`	module	`benchflow.eval_artifacts.materialize_canonical_job(output_dir)`
`--retry-from-checkpoint`	sdk	`benchflow.EvaluationConfig.retry_from_checkpoint`	needs checkpoints
`--retry-prompt`	sdk	`benchflow.EvaluationConfig.retry_prompt`
`--retry-resume-session`	sdk	`benchflow.EvaluationConfig.retry_resume_session`
`--max-cost-usd`	sdk	`benchflow.Budget.max_cost_usd`	`Evaluation(budget=Budget(...))`
`--max-sandbox-seconds`	sdk	`benchflow.Budget.max_sandbox_seconds`
`--max-tokens`	sdk	`benchflow.Budget.max_tokens`
`--max-rollouts`	sdk	`benchflow.Budget.max_rollouts`
`--token-budget` (0.10)	sdk	`benchflow.TrialBudget.tokens`	`EvaluationConfig(trial_budget=TrialBudget(...))`; a task's agent.token_budget
`--turn-budget` (0.10)	sdk	`benchflow.TrialBudget.turns`	as --token-budget; a task's agent.turn_budget
`--retry-policy`	cli		reserved: only recorded in the --run-config-out file
`--retry-attempts`	sdk	`benchflow.RetryConfig.max_retries`	EvaluationConfig.retry
`--retry-concurrency`	cli		reserved: only recorded in the --run-config-out file
`--publish-hf`	module	`benchflow.publish.huggingface.publish_folder_to_hf(repo_id)`
`--hf-prefix`	module	`benchflow.publish.huggingface.publish_folder_to_hf(path_in_repo)`
`--hf-public-read-check`	module	`benchflow.publish.huggingface.publish_folder_to_hf(public_read_check)`
`--publish-bucket`	module	`benchflow.publish.huggingface.publish_folder_to_bucket(bucket_id)`
`--eval-results-model`	module	`benchflow.publish.huggingface.open_eval_results_pr(model_repo)`
`--eval-results-dataset`	module	`benchflow.publish.huggingface.open_eval_results_pr(dataset_id)`
`--eval-results-task`	module	`benchflow.publish.huggingface.open_eval_results_pr(task_id)`
`--matrix`	sdk	`benchflow.Evaluation.__init__(config)`	one Evaluation per model entry, each with its own jobs_dir
`--trials`	sdk	`benchflow.Evaluation.__init__(job_name)`	run the Evaluation once per trial with a distinct job_name
`--n-tasks`	sdk	`benchflow.EvaluationConfig.n_tasks`
`--sample-seed`	sdk	`benchflow.EvaluationConfig.sample_seed`
`--timeout-multiplier`	sdk	`benchflow.EvaluationConfig.timeout_multiplier`
`--extra-instruction`	sdk	`benchflow.EvaluationConfig.extra_instruction`
`--dry-run`	cli		prints the resolved plan and task selection; nothing runs
`--seeds`	sdk	`benchflow.EvaluationConfig.seeds`	a list of ints; benchflow.embodied.rollouts.parse_seeds reads the CLI form

`bench eval branch`

Flag	Kind	Python	Note
`--tasks-dir`	sdk	`benchflow.branch(task_path)`	bf.branch runs one task per call
`--agent`	sdk	`benchflow.branch(agent)`
`--model`	sdk	`benchflow.branch(model)`
`--reasoning-effort`	sdk	`benchflow.branch(reasoning_effort)`
`--harness`	sdk	`benchflow.branch(harness)`
`--sandbox`	sdk	`benchflow.branch(sandbox)`
`--prompt`	sdk	`benchflow.branch(prompts)`
`--checkpoint-after-prompt`	sdk	`benchflow.branch(checkpoint_after)`
`--child`	sdk	`benchflow.branch(children)`	a label -> prompt mapping or ChildSpec items
`--snapshot-layers`	sdk	`benchflow.branch(snapshot_layers)`
`--parent`	sdk	`benchflow.branch(parent)`
`--retain-snapshots`	sdk	`benchflow.branch(retain_snapshots)`
`--from-checkpoint`	sdk	`benchflow.branch(from_checkpoint)`
`--fork`	sdk	`benchflow.branch(fork)`
`--checkpoints`	sdk	`benchflow.branch(checkpoints)`	a CheckpointPolicy: `benchflow.checkpoints.parse_checkpoint_policy()`
`--checkpoint-keep`	module	`benchflow.checkpoints.parse_checkpoint_policy(keep)`
`--resume-session`	sdk	`benchflow.branch(resume_session)`
`--child-retries`	sdk	`benchflow.branch(child_retries)`
`--stop-on-child-failure`	sdk	`benchflow.branch(continue_after_child_failure)`
`--concurrency`	sdk	`benchflow.branch(concurrency)`
`--isolate-children`	sdk	`benchflow.branch(isolate_children)`
`--agent-env`	sdk	`benchflow.branch(agent_env)`
`--include`	sdk	`benchflow.branch(task_path)`	call bf.branch once per selected task
`--jobs-dir`	sdk	`benchflow.branch(jobs_dir)`
`--job-name`	sdk	`benchflow.branch(job_name)`

`bench eval metrics`

Flag	Kind	Python	Note
`--benchmark`	module	`benchflow.metrics.collect_metrics(benchmark)`
`--agent`	module	`benchflow.metrics.collect_metrics(agent)`
`--model`	module	`benchflow.metrics.collect_metrics(model)`
`--json`	sdk	`benchflow.Job.solve_rates`	`collect_metrics(...).summary()` plus `bf.load_job(dir).solve_rates().to_dict()`
`--k`	sdk	`benchflow.Job.solve_rates(ks)`
`--solve-threshold`	sdk	`benchflow.Job.solve_rates(solve_threshold)`
`--include-controls`	sdk	`benchflow.Job.solve_rates(include_controls)`

`bench eval compare`

Flag	Kind	Python	Note
`--json`	sdk	`benchflow.Comparison.to_json`
`--labels`	sdk	`benchflow.compare(labels)`
`--vary`	sdk	`benchflow.compare(vary)`
`--on-mismatch`	sdk	`benchflow.compare(on_mismatch)`
`--include-controls`	sdk	`benchflow.compare(include_controls)`
`--attempts`	sdk	`benchflow.load_job(attempts)`	load each side, then compare
`--out`	sdk	`benchflow.Comparison.to_json(path)`
`--by`	sdk	`benchflow.compare(by)`	comma-separated on the CLI, a tuple in Python
`--k`	sdk	`benchflow.compare(ks)`
`--solve-threshold`	sdk	`benchflow.compare(solve_threshold)`

`bench eval regrade`

Flag	Kind	Python	Note
`--tasks-dir`	sdk	`benchflow.regrade(tasks_dir)`
`--sandbox`	sdk	`benchflow.regrade(sandbox)`
`--concurrency`	sdk	`benchflow.regrade(concurrency)`
`--reason`	sdk	`benchflow.regrade(reason)`
`--json`	sdk	`benchflow.RegradeSummary.to_dict`	also written to regrade-summary.json
`--review` (0.10)	sdk	`benchflow.regrade(review)`
`--reviewer-agent` (0.10)	sdk	`benchflow.ReviewerConfig.agent`	`benchflow.regrade(reviewer)`; set fields replace the trial's recorded ones
`--reviewer-model` (0.10)	sdk	`benchflow.ReviewerConfig.model`
`--reviewer-reasoning-effort` (0.10)	sdk	`benchflow.ReviewerConfig.reasoning_effort`
`--reviewer-sandbox` (0.10)	sdk	`benchflow.ReviewerConfig.environment`
`--reviewer-timeout-sec` (0.10)	sdk	`benchflow.ReviewerConfig.timeout_sec`
`--reviewer-idle-timeout` (0.10)	sdk	`benchflow.ReviewerConfig.idle_timeout_sec`
`--reviewer-storage-mb` (0.10)	sdk	`benchflow.ReviewerConfig.storage_mb`
`--reviewer-memory-mb` (0.10)	sdk	`benchflow.ReviewerConfig.memory_mb`
`--reviewer-setup-timeout` (0.10)	sdk	`benchflow.ReviewerConfig.setup_timeout_sec`
`--reviewer-evidence-timeout` (0.10)	sdk	`benchflow.ReviewerConfig.evidence_timeout_sec`
`--reviewer-concurrency` (0.10)	sdk	`benchflow.ReviewerConfig.concurrency`
`--reviewer-image` (0.10)	sdk	`benchflow.ReviewerConfig.image`
`--reviewer-agent-env` (0.10)	sdk	`benchflow.ReviewerConfig.agent_env`
`--reviewer-open-network` (0.10)	sdk	`benchflow.ReviewerConfig.open_network`

`bench train convert`

Flag	Kind	Python	Note
`--out`	module	`benchflow.trajectories.export_prime_sft.export_prime_sft_jsonl(out)`	same parameter on the TRL and branch-tree exporters
`--format`	module	`benchflow.trajectories.export_prime_sft.export_prime_sft_jsonl`	one function per format: prime-sft benchflow.trajectories.export_prime_sft.export_prime_sft_jsonl, trl-sft benchflow.trajectories.export_trl_sft.export_trl_sft_jsonl, branch-tree benchflow.trajectories.export_branch.export_branch_jsonl
`--pairs`	module	`benchflow.trajectories.export_branch.export_branch_jsonl(pairs_out)`
`--pairs-any-request`	module	`benchflow.trajectories.export_branch.export_branch_jsonl(any_request_pairs)`
`--include-oracle`	module	`benchflow.trajectories.export_branch.export_branch_jsonl(include_oracle)`
`--min-reward`	module	`benchflow.trajectories.export_prime_sft.export_prime_sft_jsonl(min_reward)`
`--row-mode`	module	`benchflow.trajectories.export_prime_sft.export_prime_sft_jsonl(row_mode)`
`--manifest`	module	`benchflow.trajectories.export_prime_sft.export_prime_sft_jsonl(manifest)`
`--expected-rows`	module	`benchflow.trajectories.export_prime_sft.export_prime_sft_jsonl(expected_rows)`
`--canonical-selection`	module	`benchflow.trajectories.export_prime_sft.export_prime_sft_jsonl(canonical_selection)`
`--redact`	module	`benchflow.trajectories.export_prime_sft.export_prime_sft_jsonl(redact)`
`--context-policy`	module	`benchflow.trajectories.export_trl_sft.export_trl_sft_jsonl(context_policy)`
`--tokenizer`	module	`benchflow.trajectories.export_trl_sft.export_trl_sft_jsonl(tokenizer_id)`
`--tokenizer-revision`	module	`benchflow.trajectories.export_trl_sft.export_trl_sft_jsonl(tokenizer_revision)`
`--max-length`	module	`benchflow.trajectories.export_trl_sft.export_trl_sft_jsonl(max_length)`
`--subagent-rows`	module	`benchflow.trajectories.export_prime_sft.export_prime_sft_jsonl(subagent_rows)`
`--reward-vector`	module	`benchflow.trajectories.export_prime_sft.export_prime_sft_jsonl(reward_vector)`	same on the TRL exporter; `benchflow.trajectories.training_signal.rollout_training_signals()` reads the vectors without converting
`--group-advantage`	module	`benchflow.trajectories.export_prime_sft.export_prime_sft_jsonl(group_advantage)`	same on the TRL exporter
`--group-by`	module	`benchflow.trajectories.export_prime_sft.export_prime_sft_jsonl(group_by)`	comma-separated on the CLI, a tuple in Python

`bench train stream`

Flag	Kind	Python	Note
`--format`	sdk	`benchflow.StreamedRollout.to_json`	jsonl is the only format; Python yields StreamedRollout objects
`--follow`	sdk	`benchflow.stream_rollouts(follow)`	--no-follow is follow=False
`--poll-interval`	sdk	`benchflow.stream_rollouts(poll_interval)`
`--timeout`	sdk	`benchflow.stream_rollouts(timeout)`	raises StreamTimeout (CLI exit 3)
`--group-size`	sdk	`benchflow.stream_rollouts(group_size)`
`--group-by`	sdk	`benchflow.stream_rollouts(group_by)`	comma-separated on the CLI, a tuple in Python
`--withheld` (0.10)	sdk	`benchflow.stream_rollouts(on_withheld)`	the CLI writes each withheld group's records to a file; Python gets the list