Terminal Bench Expert

Department Icon Digital Marketing/Marketing
Posted: 5 months ago
3-10 years
India
Digital Marketing/Marketing
work from office
Full Time

Posted: 5 months ago
This job's closing date (29 Jul 2026) has passed, so the role may already be filled. See live similar jobs.
Let Jobaaj One apply for you

An AI agent that applies to roles like this on employers' own sites. 25 applications free, no card needed.

Job Description
About Company
Similar Jobs
Please verify your account first! Send OTP

Job Description

Terminal Bench Expert

Employment Type

Contractor assignment (no medical/paid leave)

Skills

  • 3-10 years of experience in software engineering or relevant domains.
  • Strong debugging, reasoning, and analytical skills

About the Role:

Looking for highly analytical engineers, researchers, and domain specialists to contribute benchmark tasks for AI agent evaluation systems (e.g., Terminal-Bench). Design realistic, technically deep tasks simulating real-world scenarios such as debugging, data corruption, infrastructure failures, and complex workflows.

What does day-to-day look like:

  • Design high-quality Terminal-Bench task ideas and specifications.
  • Develop complex tasks requiring reasoning, investigation, and debugging.
  • Write clear task descriptions, solution approaches, and verification logic.
  • Define deterministic, outcome-based evaluation criteria.
  • Identify realistic failure modes, edge cases, and operational constraints.
  • Create tasks that challenge AI systems while remaining solvable by experts.
  • Collaborate with reviewers to refine task quality and difficulty.
  • Contribute expertise across one or more specialized domains.

Required Skills:

  • 3–10 years of experience in software engineering or relevant domains.
  • Strong debugging, reasoning, and analytical skills.
  • Good understanding of system design, workflows, and dependencies.
  • Ability to analyze complex systems across multiple layers.
  • Experience with production systems, pipelines, or large-scale workflows.
  • Strong technical writing and documentation skills.
  • Exposure to LLMs, agentic systems, or AI evaluation frameworks.
  • Experience reviewing technical specifications or designing validation logic.

Domains (Any of the following):

  • Software Engineering & Code Operations
  • Debugging & Codebase Navigation
  • System Administration & Shell Workflows
  • File & Text Processing Pipelines
  • Data Engineering (ETL & Data Pipelines)
  • Database & SQL Operations
  • Machine Learning Pipelines & MLOps
  • Post-training & Model Finetuning Workflows
  • AI Evaluation & Benchmarking Systems
  • Retrieval, Search & Ranking Systems
  • GPU / Systems Performance Optimization
  • Distributed Systems & Infrastructure
  • Cloud & Platform Engineering
  • DevOps & CI/CD Systems
  • Build & Dependency Management
  • Scientific & Numerical Computing
  • Simulation & Optimization Systems
  • Formal Methods & Theorem Proving
  • Document & Structured Data Processing (PDFs, Excel, etc.)
  • Media Processing (Video, Audio, Images via CLI tools)
  • Programmatic Graphics & Design (SVG, layout, rendering)
  • Data Visualization & Reporting Workflows
  • Geospatial & Spatial Data Processing
  • Time-series & Forecasting Systems
  • Security, Forensics & Reverse Engineering
  • Cybersecurity & Vulnerability Analysis
  • Networking & API Integration Workflows
  • Automation & Multi-step Toolchain Orchestration
  • CLI Tooling & Developer Tool Workflows
  • Version Control & Git Workflows
  • Observability, Logging & Monitoring
  • Storage Systems & File Systems
  • Finance & Accounting Workflows
  • Quantitative Finance & Risk Modeling
  • Legal & Compliance Workflows
  • Healthcare & Clinical Data Processing
  • Supply Chain & Logistics Operations
  • Marketing & Growth Analytics
  • CRM & Sales Operations
  • HR & Recruiting Analytics
  • Consulting & Strategy Modeling
  • Investment Workflows
  • Operations Research & Decision Optimization
  • Benchmark Infrastructure, Adapters & Harness

Evaluation Process (approximately 45 mins) :

  • One round of technical evaluation (45 mins)

Skills

MediaOptimizationMarketing

If a job posting appears fraudulent, asks for payment, contains misleading information, or violates our guidelines, please report it immediately. Our team will review it promptly, Jobaaj does not charge any fee from the applicants.

About Company

At Codefeast, we specialize in outsourcing top-tier Indian talent to multinational companies (MNCs).We are currently seeking a highly motivated individual with 1-2 years of experience as a Technical Project Manager to join our team as an Technical Delivery Manager. This role involves managing and coordinating our development team to ensure high-quality technical delivery, maintaining timelines, and aligning with client expectations. The ideal candidate should possess strong leadership skills, a solid technical background, and a passion for delivering excellence.

Read More

Similar Jobs

View all
Loading...

Looking for something similar?

Bag Logo
Jobaaj
Get jobs like this by email

New Digital Marketing/Marketing roles in India.

Job Alert
Google hiring for Specific Roles Apply Now!
1 min ago
New Opportunity
Amazon is hiring freshers Apply Now!
5 min ago
Featured Jobs
Microsoft opening 50+ positions Apply Now!
10 min ago

One email a day. Unsubscribe any time.