Are AI agents good physicists?
FrontierPhysics: Benchmark how AI agents do frontier physics research.
A merged task earns 4 points, a review earns 1. At 12 you are a co-author.
What a task looks like
A native BenchFlow task.md package. The prompt describes an outcome and never names a skill. The oracle must pass with reward 1.0 before any agent runs.
Prompts and oracle logic are human-authored.
Read the contributor guidetasks/<task-id>/
task.md # prompt + metadata
environment/
Dockerfile # frozen environment
skills/ # mentor skills
oracle/
solve.sh # must reach reward 1.0
verifier/
test.sh
test_outputs.py # checks the scienceExample tasks
Each one comes from research a contributor had already done.
heterodyne-shot-noise-analysis
Analyze balanced-heterodyne data. First do a shot noise analysis, then try to distinguish a weak coherent state from a vacuum state using balanced heterodyne detection traces. The data are caputured from a Thorlabs…
surface-ion-trap-shuttling
In `/root/surface_trap.stl` you have a trap model file for a surface ion trap (surface Paul trap). You need to simulate and calculate the trap frequency along the radial direction for a singly charged `40Ca+` ion when an…
trapped-ions-heating-rate
In the experiment we shot 729 nm beam to drive the transition from S to D state of 40Ca+ ions chain (9 ions chain in a linear paul trap). The trap is a marco 3D trap and 9 ions are lined up along the axial direction. We…
Earn 12 points, become a co-author
Co-authorship on the FrontierPhysics paper and dataset. Get there with 3 authored tasks, by reviewing, or any mix that adds up.
- +4
- A task you authored is merged
- +1
- A task you reviewed is merged
- 12
- Co-authorship on the paper and dataset