01 / 06
Idea
1:1 with the admin team
Before you submit a pull request, meet 1:1 with the admin team, so your PR can be reviewed in time.
Earn 12 points, become a co-author
- A merged task you authored
- 6points
- A high quality peer-review of tasks / failure mode analysis
- 2points
- Referring a contributor, once their first task merges
- 2points
Submitting a task is a lot of work, so we recommend 1 task + 3 peer reviews: 6 + 3 × 2 = 12 points.
From the team behind SkillsBench, 280+ citations since release.
01
Timeline
We will submit v1.0 with 100 tasks to Nature.
- v1.0
Get tasks merged by 31 December to join the author list of the draft submitted to Nature. Earn 12 points, become a co-author.
02
From Idea to Merge
02 / 06
Pull request
Submit the task package as a PR, using tasks that have been merged as references and following the PR template in the repository. Make sure it is:
- Human-written: task.md written by you, and rubrics drawn from human experts' domain knowledge
- Real: from realistic research you have personally done
- Challenging: hard enough for SOTA agents
03 / 06
General task review
Reviewers check the task as a whole:
- Format: the task package follows the required format
- Real research: the task comes from real research
- AI-writing detection: a reference only; reviewers decide by checking manually
- SOTA pass rate: measured once the format and everything else look good
04 / 06
Domain-expert peer review
Domain experts peer-review the task. Authors whose task has passed general review qualify to review other authors' tasks, and each high-quality review earns 2 points.
05 / 06
Failure-mode meeting
1:1 with the admin team
Once the SOTA agent runs are done and the peer review looks good, meet 1:1 with the admin team again to discuss the failure modes.
06 / 06
Merge
When every step above is finished, the task is merged, earning its author 6 points.
03
What Makes an Ideal Task
Your own work
Real research you carried out, not a problem invented for the benchmark.
Did I do this myself?
Weeks of effort
At least two weeks of genuine effort, with or without an agent helping.
Is it challenging (to SOTA agents)?
Publishability
With rubrics and verifiers, simulate how a peer researcher would audit and review the research results.
How to simulate a peer-reviewer?
04
Key Parts of a Task
task.md
The task description, in three parts.
- 1Research: Reviewing literature and making plans. The plan and thinking process are evaluated based on rubrics written by the contributor.
- 2Implementation: The concrete problem-solving request that can be verified by code scripts.
- 3Deliverables: The final output files and results ready for peer review, including paper.pdf and other deliverables.
rubric.json
The item-by-item list of rubrics that describe the expectations from the researchers. It serves as a peer-reviewer auditing the final paper, agent trajectories, etc., including research and planning parts that are not verifiable via code scripts, and also behavior alignment.
verifier
The verifier logic for checking the agent's final deliverables. Code scripts for checking verifiable results.
oracle
The ground truth answer provided by the contributor. Always gets reward == 1 on the verifier.
05
The Final Submission
- 01
A Pull Request with the task content
Open a pull request against the main branch. One task per PR, adding only files under tasks/<task-id>/.
- 02
A detailed PR description
What the original work was, the physics it exercises, and where the data came from — plus a table reporting its history against these minimums.
Report Minimum Project time scale — start and end date 2 weeks Actual working hours spent exploring the task 40 hours Estimated hours for a first-year PhD to reproduce it 10 hours - 03
A local test results report
What you ran and what happened, across multiple trials rather than a single run. Example task: PR #109 in the task repository.
- The oracle passes with reward exactly 1.0
- Results for a state-of-the-art agent, over multiple trials