Computational Science Biology Expert For AI Benchmarking

A fast-growing AI company is building expert-level benchmarks that test whether AI agents can complete real scientific workflows from start to finish, with no human help. They need working biologists to supply the hard part: realistic analyses that only someone who has done the work would know how to set up, solve, and verify.

You will turn analyses from your own research or professional work into challenging, terminal-based tasks. Each task comes with a working reference solution and an automated check that proves whether the agent got the right answer. If you have ever thought an AI could never do this part of my job, this project is about testing that claim properly.

This is a remote, task-based contract. You are paid per approved task (see How to Apply for compensation details), and you choose how many tasks you take on.

What Youll Do

• Design expert-level tasks based on real analyses you have performed, such as variant annotation, differential expression, phylogenetics, structural analysis, or pipeline debugging, that an AI agent must complete end to end in a terminal.
• Specify each task precisely: the problem statement, input files and data, software environment, and the exact deliverable the agent must produce.
• Write a complete reference solution in Python, R, or shell that shows the task can be solved in roughly 60–75 minutes of focused work.
• Build automated verification, such as an output matching a reference variant catalogue or a value falling within tolerance, and document how common mistakes would be caught.
• Package tasks reproducibly using Docker and Git so they run cleanly and consistently from the command line.
• Refine tasks based on reviewer feedback, tuning difficulty, clarity, and correctness so each one is realistic, unambiguous, and genuinely hard for todays AI agents.
• Ground every task in authentic practice, using named tools, datasets, and domain conventions such as Ensembl VEP, Bioconductor packages, or HPC workflows.

What You Bring

• PhD (completed or advanced candidacy) or 3+ years of professional research in a biology subfield, with publications, a thesis, or tangible lab or computational output.
• Hands-on analytical work within the last 3 years, and the ability to name the specific tools, datasets, and rules you used.
• You run analyses yourself in code (Python, R, C++) or directly in domain software, rather than only managing, teaching, or reviewing them.
• A clear sense of what a checkable result looks like in your field and how you would catch an error.
• Comfort with Python (or a similar language), the command line, and working with files, CSV/JSON, APIs, or databases.
• Availability to start immediately (from early November 2026) and work on a per-task basis.

Nice to Have

• First-author papers with released code or data
• Contributions to scientific or open-source bioinformatics software
• Experience with Docker, Git, and writing automated tests
• Teaching or peer-review experience in your subfield
• Experience running workloads on HPC clusters