Cyber-Index: one scale for AI cyber capability

Cyber is one of the areas where frontier models are moving fastest, and labs now tie their safeguards to it: OpenAI rated GPT-6 Astra Critical in cybersecurity, its first model to reach that level. Yet the public evidence is scattered. Each lab reports its own suites with its own harness, open-weight models are tested by other groups on other benchmarks, and many published numbers cannot be compared at all. There is no common scale.

Cyber-Index is our attempt at one. We linked 29 offensive-cyber benchmarks, from public suites like Cybench to the in-house tests of labs and government evaluators, and ran one statistical model across them that puts 168 models on a single scale. Then we reran the fit 250 times on resampled evidence, so every placement comes with an interval that shows how sure it is.

What we found

• The closed frontier leads, on its own numbers. GPT-6 Astra (80%) and Claude Opus 5.5 (76%) top the scale, and Astra is ahead in 88% of resamples. We found no third-party results for either yet: both are placed from their own lab’s evaluations, tied to the rest of the scale through older models that others did test.
• Open-weight models trail by 20 to 25 points. Kimi K3 (56%), DeepSeek V4 Flash (53%) and GLM-5.3 (52%) lead the open models. NIST’s CAISI called GLM-5.3 the most cyber-capable open-weight model as of September; on the benchmarks that test it and Kimi K3 side by side, the two are close.
• The scale agrees with the labs’ own calls. The labs’ judgements are held out of the fit and used as a test: OpenAI’s cyber risk levels line up with the index (rank correlation 0.90), and it reproduces all 20 explicit statements from OpenAI and Anthropic that one model is more cyber-capable than another.

How it works

Benchmarks are linked wherever one evaluator ran several models the same way, and an item-response model, the method behind Epoch’s Capabilities Index, puts them on one scale. We only use tables where every model ran with the same harness and budget. Leaderboards where each submitter brings its own agent, developer tables that copy competitors’ numbers, and runs blocked by safety refusals are kept out, with the reason recorded for each result.

It is an estimate from public numbers, with limits. Many models are measured on only a few benchmarks, so their intervals are wide. Benchmarks sometimes disagree. And some suites are saturated or contaminated: ExploitBench is swapped for its post-cutoff twin because OpenAI flags its results as possibly inflated by contamination.

Get your model measured

If you build models or cyber agents and want them measured and placed on the index, talk to us. The same goes if you spot a result we missed or got wrong. The full index, with every model, every source and the reasoning behind each number, is available on request.

Email xiangyi@benchflow.ai, book a call, or find us on Discord.