# Usage and telemetry > What BenchFlow records for every model call (tokens, cost, timing, settings), where each number comes from, and what happens when an agent exports no OpenTelemetry. Docs index for agents: https://www.benchflow.ai/docs/llms.txt For every trial, BenchFlow records the tokens and cost the agent used, how long each step took, and, for each model call, what was sent and what came back. It reads these numbers at the model boundary, not from the agent's own telemetry, and records where each number came from. Runs here use MiMo (`-m xiaomi/mimo-v2.6-flash`): export its key and base URL first ([Quick start](https://www.benchflow.ai/docs/benchflow/start/quick-start.md#on-another-provider)), or use `-m sonnet` with a Claude login. ## What a trial records `result.json` sums the trial's usage in its `agent_result` block, with where each number came from. From the quick start's Claude Code trial on MiMo: ```json { "n_input_tokens": 499812, "n_output_tokens": 11934, "n_cache_read_tokens": 453760, "total_tokens": 511746, "cost_usd": null, "usage_source": "provider_response", "price_source": null } ``` and from an OpenCode trial on DeepSeek, a model BenchFlow has a price for: ```json {"total_tokens": 247840, "cost_usd": 0.00857596, "usage_source": "provider_response", "price_source": "litellm"} ``` `timing.json` has seconds per step, including how much of the agent's turn was spent waiting for the model (`provider_wait`) and how much doing anything else (`agent_without_provider_wait`); [How BenchFlow works](https://www.benchflow.ai/docs/benchflow/start/how-it-works.md#one-rollout-step-by-step) shows a whole one. Each model call is a line in `trajectory/llm_trajectory.jsonl`: the request as sent, the response, its HTTP status and how long it took. The first call of the same Claude Code trial: ```text path: /v1/messages | duration_ms: 8025.3 | status: 200 request: model "mimo-v2.6-flash", reasoning_effort, stream true, tools [...] metadata: {"request_model": "mimo-v2.6-flash", "provider_model": "mimo-v2.6-flash", "call_type": "anthropic_messages", "call_purpose": "agent"} usage: {"completion_tokens": 169, "prompt_tokens": 15230, "total_tokens": 15399, "reasoning_tokens": 13, "cached_tokens": 15168} ``` That record is what to check when a setting matters: it shows the effort, sampling and model each request carried, not just what you asked for. A line holds `request.body` (the request as sent: `model`, `reasoning_effort`, `temperature`, `top_p`, `max_tokens` when set), `response.status_code`, `response.body` (`model` as answered, `choices[0].finish_reason`, `usage`) and `metadata.call_purpose`. [`requests.py`](https://github.com/benchflow-ai/benchflow/blob/v0.10.0rc3/docs/examples/audit/requests.py) prints those for every request of a job; for the Claude Code trial on MiMo in [OpenTelemetry](#opentelemetry) below: ```bash python3 requests.py jobs/2026-11-05__18-12-19 ``` ```text trial call purpose model answered effort temperature top_p max_tokens status finish primes__c02bbd43 1 agent mimo-v2.6-flash mimo-v2.6-flash high - - - 200 tool_calls primes__c02bbd43 2 agent mimo-v2.6-flash mimo-v2.6-flash high - - - 200 tool_calls primes__c02bbd43 3 agent mimo-v2.6-flash mimo-v2.6-flash high - - - 200 stop ``` Claude Code sent effort `high` on each call and no sampling settings; an OpenCode trial on DeepSeek V4 Flash sent neither, and its first call has the purpose `title`, a request OpenCode makes for itself. ## Where each number comes from `usage_source` says how the trial's tokens were counted: | `usage_source` | Means | |---|---| | `provider_response` | read from each model response, as the calls passed through BenchFlow's model proxy | | `agent_native_acp` | reported by the agent itself over ACP, as with a subscription login, which the proxy doesn't route | | `unavailable` | not recorded; treat token and cost figures for the trial as missing, not zero | `price_source` says where the price came from: `litellm`, the price table of LiteLLM, the library BenchFlow's model proxy runs on; `agent_session_log`, an estimate from the agent's own session log, as for Claude Code on a subscription login; or `null` when there's no price for the model. The proxy is always used for agents it can route, so usage and `llm_trajectory.jsonl` are always captured for them; `--usage-tracking required` additionally makes trusted usage (`provider_response` or `agent_native_acp`) a requirement of the run. ## OpenTelemetry BenchFlow 0.9.2 neither sends nor receives OpenTelemetry. Usage, cost and timing come from the model proxy and the trial's own records, so they don't depend on whether an agent exports telemetry of its own. The gap is a subscription login, which bypasses the proxy: then only what the agent reports over ACP is recorded (`agent_native_acp`), or nothing (`unavailable`). 0.10.0rc2 writes each trial's trace as OTLP/JSON to `trajectory/otel-spans.json`, sends it to the collector `OTEL_EXPORTER_OTLP_ENDPOINT` names where `bench` runs, and keeps the exports of an agent whose telemetry you turn on in `trajectory/agent-otel.jsonl` ([OpenTelemetry in the model-call records](https://github.com/benchflow-ai/benchflow/blob/v0.10.0rc3/docs/reference/model-calls.md#opentelemetry)). On 0.9.2, an agent can still send its own telemetry to a receiver you run. Claude Code's is off until you turn it on with its variables, which `--agent-env` passes into the sandbox. The receiver must listen on an address the sandbox can reach: for local Docker on Linux, Docker's bridge, which the first line below reads. Docker Desktop on macOS or Windows has no `docker0`; there a container reaches the host as `host.docker.internal`, so run the receiver on `0.0.0.0` and use that name in the endpoint (we ran this on Linux). [`receive.py`](https://github.com/benchflow-ai/benchflow/blob/v0.10.0rc3/docs/examples/otel/receive.py) is a receiver of about 30 lines that keeps every export as a JSON line: ```bash git clone --depth 1 --branch v0.10.0rc3 https://github.com/benchflow-ai/benchflow.git mkdir -p tasks && cp -r benchflow/docs/examples/primes tasks/ cp benchflow/docs/examples/otel/receive.py benchflow/docs/examples/otel/calls.py . BRIDGE=$(ip -4 -o addr show docker0 | awk '{print $4}' | cut -d/ -f1) python3 receive.py $BRIDGE 4318 otlp.jsonl & bench run tasks/primes -a claude -m xiaomi/mimo-v2.6-flash \ --agent-env CLAUDE_CODE_ENABLE_TELEMETRY=1 \ --agent-env OTEL_LOGS_EXPORTER=otlp \ --agent-env OTEL_METRICS_EXPORTER=otlp \ --agent-env OTEL_EXPORTER_OTLP_PROTOCOL=http/json \ --agent-env OTEL_EXPORTER_OTLP_ENDPOINT=http://$BRIDGE:4318 \ --agent-env OTEL_LOGS_EXPORT_INTERVAL=1000 ``` The trial passed in 31 seconds, and the receiver got five log exports and one metrics export. Claude Code logs an `api_request` event for each model call, with its tokens. [`calls.py`](https://github.com/benchflow-ai/benchflow/blob/v0.10.0rc3/docs/examples/otel/calls.py) prints them beside the same calls in the proxy's record: ```bash python3 calls.py otlp.jsonl jobs/2026-11-05__18-12-19/primes__c02bbd43 ``` ```text call otel: input cache_read output proxy: prompt_tokens completion_tokens 1 35 14720 169 14755 169 2 215 14720 91 14935 91 3 125 14912 47 15037 47 ``` The two records agree call by call once you know how each counts. Claude Code's `input_tokens` leave out the tokens read from the cache, and the proxy's `prompt_tokens` include them, so input plus cache_read is the prompt. Output tokens are the same number in both. Each event also carries a `cost_usd`: that is Claude Code's own estimate from its price table, which isn't MiMo's price, so price a non-Claude model's tokens yourself, as below. The trial's `config.json` keeps every variable you pass with `--agent-env`, values included, under `agent_env`. BenchFlow leaves out a variable whose name looks secret (it contains `KEY`, `TOKEN`, `SECRET`, `PASSWORD`, `AUTH` or `SESSION`, among others) or whose value looks like a credential. ## Cost, and models without a price BenchFlow prices tokens with LiteLLM's table. In 0.9.2 the proxy downloads that table from GitHub when it starts, if it can, and otherwise reads the copy installed with it, so the same run can be priced differently on another day. 0.10.0rc2 reads only the installed copy and carries vendors' own rates for 18 current models, MiMo V2.6 and DeepSeek V4 among them. Two consequences for 0.9.2: - **DeepSeek V4's dollar figures are placeholders.** Its routes are priced at an unverified $0.28 and $0.42 per million input and output tokens, not DeepSeek's rates, so the DeepSeek costs in these docs, such as the $0.00857596 above, are that placeholder's. Report DeepSeek's tokens, or price them from DeepSeek's rate card with the date you read it. - **A model missing from the table gets no price**, such as MiMo or any model newer than the table: `cost_usd: null`, and the run summary says so: ```text Cost: not reported in USD (subscription logins and unpriced models report tokens only), 511,746 tokens ``` For such models, compare runs by tokens, cap them with `--max-tokens` rather than `--max-cost-usd`, and price the tokens yourself from the provider's rate card when you report a cost.