BenchFlow
News

Research

How we improved SkillsBench v1.1 scores by 69.4% using env0

Bingran YouBingran You

What is env0

LLM agents are increasingly deployed to automate productivity tasks — email triage, meeting scheduling, document management — but evaluating them on live services is risky due to potentially irreversible changes. Existing benchmarks rely on simplified environments and fail to capture realistic, stateful, multi-service workflows. So we made env0 to address this. env0 inherits from the mock environments in ClawsBench, a benchmark for evaluating and improving LLM agents in realistic productivity settings. The current v0.1 version has five high-fidelity mock services that replicate real productivity and messaging APIs with full state management and deterministic snapshot/restore.

We run post-train experiments to demonstrate how env0 can be used to improve model performance.

Configuration

env0 → SFT distillation: sample 300 tasks from the env0-mobile dataset, then Qwen3.5-397B-A17B teacher rollouts produce 300 SFT trajectories to fine-tune Qwen3.5-9B.

The entire experiment was completed on a single H100 80GB GPU.

The experiment uses the env0-mobile dataset — 2,003 verified productivity tasks, each paired with an executable oracle verifier.

As proof-of-concept experiment, we use Qwen3.5-397B-A17B as teacher model and Qwen3.5-9B as student model. We randomly picked 300 tasks from the env0-mobile dataset and use Qwen3.5-397B-A17B + OpenHands as teacher model to generate teacher model trajectories. In these 300 tasks, Qwen3.5-397B-A17B gets reward == 1 in 63 tasks, pass rate = 21.0%. We keep the full 300 trajectories set for SFT training. We use BenchFlow as eval harness, which can by default generate results.jsonl formatted trajectories that is compatible with prime-rl. BenchFlow also natively supports using prime-rl in BenchFlow CLI and SDK as post-train framework.

SettingValue
StudentQwen3.5-9B (non-quantized, BF16)
Teacher (SFT-data source)Qwen3.5-397B-A17B
TrainerBring-your-own SFT-data path, 1× H100 80GB
Renderer / chat templateQwen3.5 chat template (chat_template.jinja)
LossSFT cross-entropy (next-token on assistant turns)
Dataset size300 env0-mobile teacher trajectories (prime-sft.jsonl)
Sequence length8192
Epochs / steps1 epoch / 300 steps
Learning rate1e-4
Batch size1 (× gradient accumulation 8)
LoRA rank32
LoRA alpha / dropout64 / 0.05
Target modulesq_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Trained scopetext / language parameters only (no multimodal)

Results

Training curves

Train loss
01234567050100150200250300StepLoss
Eval loss
012345678050100150200250300StepLoss
GPU utilization
020406080100050100150200250300StepPercent
GPU memory used
0102030405060050100150200250300StepGiB
SFT training metrics — Qwen3.5-9B · BF16 LoRA · 8k context · H100 · 300 steps.

Improvement on ClawsBench

After only 300 training steps, the SFT version of Qwen3.5-9B (open source on HF: huggingface.co/benchflow/benchflow-qwen35-9b) shows 31.58% performance improvement in 60 held-out eval tasks set (trial = 3, pass rate goes from 10.56% to 13.89%). For model inference we use SGLang to host Qwen3.5-9B both before and after SFT.

015304560Pass rate (%)10.56%13.89%43.33%51.67%60 held-out eval tasksstrict pass: reward ≥ 1.0+31.58%Qwen3.5-9BQwen3.5-9B SFT (300 steps)Qwen3.5-397B-A17B (teacher)GPT-5.4-minienv0 Standard60 strict pass rateAll models use OpenHands as the agent harness
Pass rate on the 60-task held-out eval, with ±1 standard-error bars (binomial). Qwen3.5-9B bars use n = 180 = 60 tasks × 3 trials; the Qwen3.5-397B-A17B teacher and GPT-5.4-mini references use n = 60 each from one standard60 run.

Improvement on SkillsBench

We also evaluate on SkillsBench (87 tasks, 8 domains) using the SFT version Qwen3.5-9B + SGLang + OpenHands + with-skill setup, where the same custom SFT snapshot lifts the pass rate from 7.01% to 11.88% — a +69.4% relative gain (1.69× baseline). These SkillsBench numbers are still provisional as the final trial completes. The full SkillsBench leaderboard is at skillsbench.ai/leaderboard.

051015Pass rate (%)7.01%11.88%87 SkillsBench taskswith-skill setting+69.4%Qwen3.5-9BQwen3.5-9B SFTSkillsBench pass rateQwen3.5-9B via SGLang + OpenHands, with skill
SkillsBench pass rate for Qwen3.5-9B before and after custom SFT, with ±1 standard-error bars (binomial). Baseline n = 870 = 87 tasks × 10 trials; SFT snapshot n = 261 ≈ 87 × 3 trials (provisional — final trial still completing).

The current results are still preliminary and we are running more ablation to explore other experiment configurations.

Cost

SFT training is cheap: a single H100 80GB ran 3.24 h at $3.29/h, $10.67 total. Excludes sandbox/runtime infra and retries.

References

  1. Qwen Team. Qwen3.5. Qwen blog. The teacher (Qwen3.5-397B-A17B) and student (Qwen3.5-9B) models used in this experiment. qwen.ai/blog?id=qwen3.5
  2. Li, X., et al. SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks. arXiv:2602.12670, Feb. 2026. The held-out agent-skills benchmark reported in the Results. arxiv.org/abs/2602.12670
  3. Li, X., et al. ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces. arXiv:2604.05172, Apr. 2026. env0 inherits its mock environments from ClawsBench. arxiv.org/abs/2604.05172
  4. Prime Intellect. General Agent. Prime Intellect blog. primeintellect.ai/blog/general-agent
  5. Prime Intellect. (2025). PRIME-RL [Computer software]. github.com/PrimeIntellect-ai/prime-rl
  6. BenchFlow team. (2026). BenchFlow: framework for RL environments for LLM agents (Version 0.6.4) [Computer software]. github.com/benchflow-ai/benchflow
  7. BenchFlow team. (2026). env0: simulated-workspace environments for post-training LLM agents (Version 0.1.0) [Computer software]. github.com/benchflow-ai/env0

Acknowledgments

We gratefully thank Prime Intellect for sponsoring the H100 GPU compute that made this experiment possible.

Data availability

The dataset is available on request; contact us to access it.

All news