An AI agent that applies to roles like this on employers' own sites. 25 applications free, no card needed.
AI summary of this job
Beta Read in 10 secondsAn AI Evaluation Engineer role at Dell Technologies in Karnataka for people with 8–15 years of experience. You'll build evaluation harnesses in Python, design measurable analytics for models and agentic tools using LangFuse, and own the release gate for AI-native security capabilities.
- Work from office
- 8–15 years
What you'll do
- Build and maintain evaluation harnesses in Python
- Design measurable analytics using LangFuse
- Own the release gate for AI-native capabilities
Good fit if you
- Have 8+ years in software engineering or machine learning
- Build evaluation harnesses in Python
- Build measurable analytics of models and agentic tools
AI-generated from the job description. Check the full description before you apply.
Job Description
AI Evaluation Engineer
The Security Resiliency Organization builds the AI-native capabilities that are replacing manual security processes across Dell, and every one of those capabilities has to earn its release through evidence, not demonstration. This team owns the quality bar for that shift: defining what "good enough" means for each workflow and building the harnesses that prove it, before and after release.
Join us as an AI Evaluation Engineer on our Security Resiliency team in Bangalore, India to do the best work of your career and make a profound social impact.
What you’ll achieve
- You will own the release gate for AI-native security capabilities, deciding what evidence is required before a workflow can go live.
- You will build evaluation harnesses that measure model and agent behaviour objectively, not by sampling outputs and forming an impression.
- You will turn evaluation into a repeatable, measurable analytics practice so that every workflow after the first costs less to evaluate than the one before it.
You will:
- Define and record acceptance criteria for each capability with the functional owner before evaluation begins, and specify the evidence, ground truth and coverage required to support the claim.
- Build and maintain evaluation harnesses in Python covering dataset construction, test orchestration, scoring and reporting, and integrate them as automated quality gates in CI/CD.
- Design measurable analytics for models and agentic tools – quantifiable metrics for accuracy, consistency across repeated runs, confidence-versus-accuracy calibration, latency and cost – using tooling such as LangFuse for trace-level evaluation and embedding-based similarity scoring for output comparison.
- Own the release gate: apply the criteria to the evidence, record the result, and block releases that do not clear the threshold, with thresholds set proportionate to the consequence of an incorrect agent action.
- Run ongoing regression evaluation after release to detect behavioural drift from model, prompt, data or upstream changes, and report results to engineering and functional stakeholders in decision-ready terms.
Take the first step towards your dream career
Every Dell Technologies team member brings something unique to the table. Here’s what we are looking for with this role:
Essential Requirements
- 8+ years in software engineering, quality engineering or machine learning evaluation, with systems shipped to production.
- Hands-on construction of evaluation harnesses in Python: dataset design, test orchestration, scoring implementation and result reporting for non-deterministic systems.
- Ability to build measurable analytics of models and agentic tools, using LangFuse (or equivalent LLM observability tooling) and embedding-based comparison techniques.
- Statistical literacy: sample size, variance across runs, and the difference between a real change in behaviour and noise, applied with Python data tooling at volume.
- Experience owning a quality gate or release decision, including declining a release and holding that position under schedule pressure.
Desirable Requirements
- Experience evaluating AI systems within a security domain, and familiarity with agent orchestration frameworks such as LangGraph to understand what is being evaluated.
- Familiarity with the OWASP Top 10 for LLM Applications and the NIST AI Risk Management Framework, ideallyin an audited or regulated environment.
Skills
PythonGoMachine LearningLLMCI/CDOWASPRisk ManagementIf a job posting appears fraudulent, asks for payment, contains misleading information, or violates our guidelines, please report it immediately. Our team will review it promptly, Jobaaj does not charge any fee from the applicants.
About Company
Similar Jobs
View allLooking for something similar?
New IT / Software Development & Related roles in Karnataka.
One email a day. Unsubscribe any time.

