Earn 12 points, become a co-author
A merged task you authored earns 6, one you reviewed earns 2, and referring a contributor earns 2 once their first task merges.
Reviewing opens up once you have your first good task merged — ask a maintainer to be added as a reviewer.
From the team behind SkillsBench, 200+ citations since release.
Timeline
We will submit to ICLR first and then submit to Nature after further polish.
- v0.1
Get tasks merged by 7 September to join the author list of ICLR (and all future paper versions).
- v1.0
Get tasks merged by 31 December to join the author list of the draft submitted to Nature.
What makes an ideal task
Your own work
Real research you carried out, not a problem invented for the benchmark.
Did I do this myself?
Weeks of effort
At least two weeks of genuine effort, with or without an agent helping.
Is it challenging (to SOTA agents)?
Publishability
With rubrics and verifiers, simulate how a peer researcher would audit and review the research results.
How to simulate a peer-reviewer?
Key parts of a task
task.md
The task description, in three parts.
- 1.Research: Reviewing literature and making plans. The plan and thinking process are evaluated based on rubrics written by the contributor.
- 2.Implementation: The concrete problem-solving request that can be verified by code scripts.
- 3.Deliverables: The final output files and results ready for peer review, including paper.pdf and other deliverables.
rubric.json
The item-by-item list of rubrics that describe the expectations from the researchers. It serves as a peer-reviewer auditing the final paper, agent trajectories, etc., including research and planning parts that are not verifiable via code scripts, and also behavior alignment.
verifier
The verifier logic for checking the agent's final deliverables. Code scripts for checking verifiable results.
oracle
The ground truth answer provided by the contributor. Always gets reward == 1 on the verifier.
The final submission
- 01
A Pull Request with the task content
Open a pull request against the main branch. One task per PR, adding only files under tasks/<task-id>/.
- 02
A detailed PR description
What the original work was, the physics it exercises, and where the data came from — plus a table reporting its history against these minimums.
Report Minimum Project time scale — start and end date 2 weeks Actual working hours spent exploring the task 40 hours Estimated hours for a first-year PhD to reproduce it 10 hours - 03
A local test results report
What you ran and what happened, across multiple trials rather than a single run. Example task: PR #23 in the task repository.
- The oracle passes with reward exactly 1.0
- Results for a state-of-the-art agent, over multiple trials