Contribute
Turn research you have already done into a benchmark task
Earn 12 points, become a co-author
A merged task you authored earns 4, one you reviewed earns 1. 3 authored tasks gets you there, as does any mix that adds up. Points land on merge.
Who should contribute
- A PhD or current PhD candidate in physics, EECS, or an adjacent field
- Or extensive hands-on experience in a physics lab or an equivalent industry role
What makes an ideal task
All three. A task that misses any one will not merge.
Your own work
Real research you carried out, not a problem invented for the benchmark.
Did I do this myself?
Weeks of effort
At least two weeks of genuine effort, with or without an agent helping.
Did it really take weeks to finish?
Verifiable
The result is right or wrong, and a script can tell which.
Can a script grade it?
One excellent task is worth more than many mediocre ones.
The four steps
- 01
Ideate
Pick a project that meets all three. Bring it to group chat or confirm with a maintainer before you build.
- 02
Create
Write the task package: prompt and metadata, Docker environment, mentor skills, oracle, verifier.
- 03
Test
Run the oracle, then at least one agent with and without skills, over multiple trials.
- 04
Submit
Fork the repository and open a pull request against main.
What you must write yourself
Use an AI assistant for the plumbing. The science has to be yours.
Never delegate
- The prompt body — written by hand, in imperative prose
- The oracle solution, deriving the answer by computation
- The scientific judgement about what counts as correct
- The claim that this reflects real research practice
Fine to delegate
- Dockerfile scaffolding and pinning dependencies
- Boilerplate for the verifier test harness
- Formatting metadata and frontmatter
- Tidying prose you have already written
The final submission
Three things.
- 01
A PR from your fork
Fork this repository and open a pull request against main here. One task per PR, touching only files under tasks/<task-id>/.
- 02
A detailed PR description
The history of the task: what the original work was, its start and end dates, and how many working hours you spent. Plus the physics it exercises and where the data came from.
- 03
A local test results report
What you ran and what happened: the oracle at reward 1.0, and agent pass rates with and without skills across multiple trials. Example task: PR #2.
Before you open it, both of these have to pass, with the oracle returning reward 1.0:
bench tasks check tasks/<task-id> bench eval run --tasks-dir tasks/<task-id> --agent oracle --sandbox docker
Give real dates and an honest hour count, and report only the runs you actually completed.
Ten minutes of triage can save a weekend
Bring your idea to Discord before you build it.