AI Benchmark Researcher (Remote)

Department Icon Data Science Analytics & Machine Learning
Posted: 6 days ago
1-3 years
India
Data Science Analytics & Machine Learning
work from office
Full Time
Closes 9 Jan 2027

Posted: 6 days ago
Let Jobaaj One apply for you

An AI agent that applies to roles like this on employers' own sites. 25 applications free, no card needed.

Job Description
About Company
Similar Jobs
Please verify your account first! Send OTP

Job Description

  • Role Title: AI Benchmark Researcher (Remote)
  • Role Type: Full-time
  • Location: Remote

Role Overview

We are seeking an analytical researcher and red-teaming specialist to build evidence-based web-research benchmark tasks that reliably expose weaknesses in state-of-the-art (SOTA) AI models (e.g., Claude Opus 5 Max). In this role, you will discover hard-to-surface facts, build precise research tasks around them, test them directly against leading LLMs, and diagnose the exact failure mechanisms in the models search and reasoning paths.

Your work will directly shape AI evaluation frameworks, ensuring next-generation frontier models are tested against rigorous, non-trivial real-world research scenarios.

Job Requirements

1. Core Experience: 1+ years in investigative research, fact-checking, AI benchmark task authoring, red-teaming, or technical QA review.

2. Benchmark Familiarity: Proven experience authoring, testing, or reviewing tasks for AI benchmarks such as GAIA, BrowseComp, SimpleQA, SWE-bench, Terminal-Bench, or Humanitys Last Exam (HLE).

3. Analytical & OSINT Skills: Intermediate-to-advanced web search and Open Source Intelligence (OSINT) skills; ability to navigate academic archives, local records, or obscure databases.

4. Model Trajectory Debugging: Strong analytical capability to inspect model reasoning chains, search logs, and source selection to diagnose root causes of failure.

5. Precision Writing: Ability to author clear, unambiguous benchmark prompts with bulletproof, primary-sourced answer keys.

Job Responsibilities:

  • 1. Obscure Fact Discovery: Deeply research niche topics to identify factual ground truths that cannot be surfaced via simple, direct search queries.
  • 2. Source Verification: Rigorously verify facts against primary or highly credible sources, cross-check for contradictory evidence, and document exact source URLs.
  • 3. Task Construction: Draft clear, answerable benchmark tasks designed specifically to expose structural weaknesses in model research strategies.
  • 4. Red-Teaming & Model Testing: Execute tasks against designated SOTA models to confirm they reliably trigger intended model failures.

5. Failure Diagnosis & Debugging: Trace model run logs, search trajectories, and intermediate outputs to pinpoint exactly where and why the model failed (distinguishing genuine model limitations from ambiguous prompts).

  • 6. Pattern Scaling & Review: Apply validated model-breaking research patterns across new domains and, for strong performers, review tasks authored by peer trainers.
  • Strong Plus (Preferred Qualifications)
  • 7. Bachelor or higher degree from relevant engineering domain is a must for this position.
  • 8. Prior experience specifically designing prompt-injection or reasoning-trap tasks that break SOTA models.
  • 9. Advanced OSINT capabilities or published investigative/academic research demonstrating deep source verification.
  • 10. Niche domain expertise in areas rich with hard-to-find records (e.g., local government filings, regional sports history, legal archives, historical patents).
  • 11. Experience serving as a lead reviewer, benchmark auditor, or quality owner on AI evaluation projects.

Offer Details:

  • 1. Commitments Required: Upto 40 hours per week with 4 hours of overlap with PST.
  • 2. Engagement type: Contractor
  • 3. Engagement Length: upto 4 weeks

Evaluation Process

1. Candidates will be require to submit a Job Interest Form & assessment.

2. Talents finalized in assessment will go through assessment, review & will proceed further accordingly.

What Youll Do

  • 1. Obscure Fact Discovery: Deeply research niche topics to identify factual ground truths that cannot be surfaced via simple, direct search queries.
  • 2. Source Verification: Rigorously verify facts against primary or highly credible sources, cross-check for contradictory evidence, and document exact source URLs.
  • 3. Task Construction: Draft clear, answerable benchmark tasks designed specifically to expose structural weaknesses in model research strategies.
  • 4. Red-Teaming & Model Testing: Execute tasks against designated SOTA models to confirm they reliably trigger intended model failures.

5. Failure Diagnosis & Debugging: Trace model run logs, search trajectories, and intermediate outputs to pinpoint exactly where and why the model failed (distinguishing genuine model limitations from ambiguous prompts).

6. Pattern Scaling & Review: Apply validated model-breaking research patterns across new domains and, for strong performers, review tasks authored by peer trainers.

Strong Plus (Preferred Qualifications)

Apply Now!

Skills

AiLlms

If a job posting appears fraudulent, asks for payment, contains misleading information, or violates our guidelines, please report it immediately. Our team will review it promptly, Jobaaj does not charge any fee from the applicants.

About Company

Joblet AI is an AI startup building an AI-powered career coach to help job seekers achieve their career goals.

Similar Jobs

View all
Loading...

Looking for something similar?

Bag Logo
Jobaaj
Get jobs like this by email

New Data Science Analytics & Machine Learning roles in India.

Job Alert
Google hiring for Specific Roles Apply Now!
1 min ago
New Opportunity
Amazon is hiring freshers Apply Now!
5 min ago
Featured Jobs
Microsoft opening 50+ positions Apply Now!
10 min ago

One email a day. Unsubscribe any time.