FrontierPhysics: Evaluating Agents for End-to-End Physics Research

182submissions56tasks9domains7modalities42researchers38institutions

See Leaderboard Contribute a Task Benchmark Collaboration

Agent Performance

Hover a point to pin its exact pass rate and wall-clock to the axes. Dashed lines mark fleet means; the emerald line is the Pareto frontier.

Failure Modes

GPT-6 Astra (xhigh)

Fully human-reviewed labels for 168 agent rollouts

One square per rollout · 56 tasks × 3 trialsRIG68 of 168

Condensed Matter

AMO

Quantum Info

Astrophysics

Fluids

Statistical

Photonics

High Energy

Chemical

Agent failuresn · % of 143 failed

Grading errors

Also in the taxonomy

Hover a tag or a square · a rollout can carry several tags, so the shares add up to more than 100%

Agent Leaderboard

Sort by

#	Agent	Verifier	Overall	Pass@3	Score	Visualization
1	Claude Opus 5.5Claude Code	40.5%	22.6%	37.5%	21.3	40.5 → 22.6%
2	GPT-6 Astra (max)Codex	34.5%	17.9%	26.8%	16.7	34.5 → 17.9%
3	GPT-6.1 SolCodex	31.0%	16.1%	25.0%	15.3	31.0 → 16.1%
4	GPT-6 Astra (xhigh)Codex	31.5%	14.9%	26.8%	14.0	31.5 → 14.9%
5	Claude Fable 5.1Claude Code	28.6%	13.1%	19.6%	12.0	28.6 → 13.1%
6	GPT-6 SolCodex	21.4%	11.3%	23.2%	10.0	21.4 → 11.3%
7	Claude Opus 5Claude Code	16.7%	10.1%	17.9%	9.5	16.7 → 10.1%
8	GPT-5.6 Sol (max)Codex	17.9%	8.3%	12.5%	7.4	17.9 → 8.3%
9	Grok 4.7Grok Build	17.9%	6.5%	14.3%	6.0	17.9 → 6.5%
10	Claude Haiku 5.5Claude Code	14.3%	4.2%	10.7%	3.5	14.3 → 4.2%
11	GPT-5.6 Luna (max)Codex	11.9%	3.6%	3.6%	2.9	11.9 → 3.6%
12	GPT-5.6 Sol (xhigh)Codex	9.5%	3.0%	7.1%	2.6	9.5 → 3.0%
13	Grok 4.6Grok Build	4.2%	3.0%	5.4%	2.6	4.2 → 3.0%
14	GPT-5.6 Terra (max)Codex	14.9%	3.0%	5.4%	2.5	14.9 → 3.0%
15	GPT-5.6 Terra (xhigh)Codex	9.5%	1.8%	3.6%	1.4	9.5 → 1.8%
16	GLM-5.3Claude Code	7.1%	1.2%	3.6%	1.1	7.1 → 1.2%
17	MiMo V2.6 ProClaude Code	9.5%	1.2%	3.6%	1.0	9.5 → 1.2%
18	GPT-5.6 Luna (xhigh)Codex	9.5%	1.2%	1.8%	0.9	9.5 → 1.2%
19	Kimi K3Claude Code	3.0%	0.6%	1.8%	0.4	3.0 → 0.6%
20	DeepSeek V4 ProClaude Code	3.0%	0.6%	1.8%	0.3	3.0 → 0.6%
21	Gemini 3.8 FlashAntigravity	7.7%	0.0%	0.0%	0.0	7.7 → 0.0%

Hover over a row to see confidence intervals, time and cost. Score is 0–100: a passing rollout's weighted rubric score, 0 for any failure.

FrontierPhysics · 17 models · 21 configurations · 56 tasks · 3 trials per task · 95% CIsOverallVerifier only

Anthropic

OpenAI

xAI

Z.ai

Xiaomi

Moonshot

DeepSeek

Google

Cost Efficiency

Hover a point to pin its exact pass rate and cost to the axes. Dashed lines mark fleet means; the emerald line is the Pareto frontier. GLM-5.3's cost is an estimate from token usage at list prices.

Physics-Domain Profile

Condensed Matter Physics12 tasks

Claude Opus 5.5Claude Code41.7 → 30.6%

GPT-6 Astra (max)Codex33.3 → 27.8%

Grok 4.7Grok Build27.8 → 16.7%

solid = overall pass · pale = passed the verifier only · hover another radar axis to switch domain

Claude Opus 5.5· Claude CodeGPT-6 Astra (max)· CodexGrok 4.7· Grok Buildrings at 20–60% · overall pass

Tasks

A figure from a FrontierPhysics task in Condensed Matter Physics — Condensed Matter Physics

A figure from a FrontierPhysics task in AMO Physics — AMO Physics

A figure from a FrontierPhysics task in Quantum Information — Quantum Information

A figure from a FrontierPhysics task in Astrophysics — Astrophysics

A figure from a FrontierPhysics task in Fluid Dynamics — Fluid Dynamics

A figure from a FrontierPhysics task in Statistical Physics — Statistical Physics

Figure credits

Contribute a Task