QA Automation Engineer / SDET AI Testing (Mohali)

Department Icon Data Science Analytics & Machine Learning
Posted: 1 week ago
2-8 years
Nagar, Rajasthan
Data Science Analytics & Machine Learning
work from office
Full Time
Closes 10 Jan 2027

Posted: 1 week ago
Let Jobaaj One apply for you

An AI agent that applies to roles like this on employers' own sites. 25 applications free, no card needed.

AI Summary
Job Description
About Company
Similar Jobs
Please verify your account first! Send OTP

AI summary of this job

Beta Read in 10 seconds

A TestOps Engineer – Agentic AI role at XenonStack Moments in Nagar for people with 2–8 years of experience. You'll build and maintain reusable evaluation harnesses for LLM-powered agents and agentic workflows, integrating offline evaluations into CI/CD release gates and validating agent trajectories and tool selection.

  • Nagar
  • Work from office
  • 2–8 years

What you'll do

  • Build reusable evaluation harnesses for LLMs
  • Integrate offline evaluations into CI/CD gates
  • Measure agent quality against thresholds

Good fit if you

  • Have 2–8 years of engineering experience
  • Know Python automation and LLM evaluation
  • Understand agentic AI and workflow testing
Sounds like you? Applying takes under a minute.

AI-generated from the job description. Check the full description before you apply.

Helpful?

Job Description

TESTOPS ENGINEER – AGENTIC AI

  • Location: XenonStack office, Sector 75, STPI, Mohali (Chandigarh Tricity), work from office
  • Experience: 2–8 years | Employment: Full time, XenonStack Private Limited | Team: AgentOps / Agentic AI Engineering
  • Role Level: X2 – Workflow

About The Role

We are looking for a TestOps Engineer – Agentic AI with strong expertise in Python automation, LLM evaluation, agentic workflow testing, and CI/CD integration.

You will design and maintain shared evaluation infrastructure, golden datasets, automated quality gates, and regression frameworks for AI agents across ElixirData, ElixirClaw, and NexaStack.

This is an engineering-focused TestOps role, not a traditional manual QA position. You will build the testing systems and enable engineering teams to test their own components.

ROLE MISSION

Make every AI agent release measurable, repeatable, reliable, and safe to ship through automated evaluations, statistical quality measurements, and continuous regression testing.

Key Responsibilities

  • Build and maintain reusable evaluation harnesses for LLM-powered agents and agentic workflows.
  • Integrate offline evaluations for prompts, models, tools, and retrieval into CI/CD release gates.
  • Measure agent quality, reliability, latency, and execution cost against defined thresholds.
  • Create and version golden datasets, synthetic conversations, and edge-case scenario libraries for customer workflows.
  • Validate agent trajectories, tool selection, execution sequences, and tool arguments.
  • Develop contract tests for MCP servers, APIs, and external tools used by agents.
  • Handle non-deterministic AI behavior using repeated evaluations, pass@k scoring, and statistical thresholds.
  • Implement LLM-as-a-Judge evaluations and calibrate results against human-labelled datasets.
  • Conduct adversarial testing for prompt injection, jailbreaks, data leakage, and policy violations.
  • Execute regression evaluations during model, prompt, provider, and dependency upgrades.
  • Establish release quality gates and validate releases against agreed quality and safety thresholds.
  • Convert production incidents into automated regression tests and maintain shared quality dashboards.
  • Provide reusable test templates and evaluation playbooks to engineering teams.

How We Work With Ai

AI is part of the daily workflow at XenonStack, not an add-on.

  • At our Mohali/Chandigarh office, AI tools are open for use: coding assistance, test generation, evaluation design, debugging, documentation, and research.
  • When working on a clients data or environment, follow the clients compliance and information security policy. Use only approved tools and never expose client data, code, credentials, or configurations to unapproved AI services.
  • Every AI-generated test, evaluation result, and recommendation must be reviewed, validated, and understood before use.
  • Come ready to discuss your AI testing stack, how you evaluate non-deterministic outputs, and how you prevent unsafe agent behavior.

What Success Looks Like
  • 100% of agent releases go through automated evaluation gates.
  • Every active customer workflow has a versioned golden dataset with agreed scenario coverage.
  • Every production incident is converted into a regression test within 5 days.
  • Zero critical AI safety findings reach production.
  • All engineering pods adopt the shared evaluation harness rather than maintaining separate testing scripts.
  • Quality dashboards provide measurable visibility into agent reliability, latency, cost, and evaluation results.

Must-have Skills
  • Python and test automation: Strong Python programming, pytest, reusable test framework development, API testing, automated regression testing, and Git.
  • LLM evaluation frameworks: Hands-on experience with DeepEval, Promptfoo, Ragas, LangSmith, or equivalent evaluation tools.
  • Agentic AI: Understanding of RAG, LLMs, tool calling, agent execution trajectories, and agent frameworks.
  • Agent integration testing: Experience validating APIs, MCP servers, tool contracts, and multi-step agent workflows.
  • Evaluation techniques: Golden datasets, synthetic test generation, pass@k, statistical thresholds, repeated runs, and LLM-as-a-Judge evaluation.
  • CI/CD and release engineering: Integrating automated evaluations into CI/CD pipelines and enforcing release quality gates.
  • AI safety testing: Knowledge of prompt injection, jailbreaks, data leakage, adversarial scenarios, and policy validation.
  • Engineering fundamentals: Debugging, root-cause analysis, test coverage, reproducibility, and maintainable automation code.

Good-to-have Skills
  • LangChain, LangGraph, or other agent orchestration frameworks.
  • Claude, Azure OpenAI, Amazon Bedrock, or other LLM providers.
  • Multi-agent system testing and agent-to-agent interaction validation.
  • Docker, Kubernetes, and cloud-native deployment environments.
  • AI observability, tracing, quality dashboards, and production monitoring.
  • AI security, red-teaming, responsible AI, and governance frameworks.
  • Experience with model and prompt versioning, benchmarking, and cost optimisation.

CANDIDATE PROFILE
  • 2–8 years of experience in QA Automation, SDET, MLOps, or related engineering roles, including hands-on LLM evaluation.
  • Proven experience developing Python-based automation frameworks and integrating them into CI/CD workflows.
  • Practical understanding of LLM applications, RAG systems, tool calling, and agentic AI architectures.
  • Ability to evaluate non-deterministic systems using statistical metrics rather than relying only on binary pass/fail checks.
  • Bachelors degree in Computer Science, Information Technology, Engineering, or a related discipline; demonstrated hands-on experience is equally important.
  • Strong analytical thinking, debugging, communication, and cross-team collaboration skills.

We value engineers who build scalable testing systems rather than simply execute test cases. Be ready to explain how you evaluated an AI agent, designed golden datasets, handled inconsistent outputs, and prevented regressions from reaching production.

Traditional manual QA experience alone is not sufficient for this role.

HIRING PROCESS

Pre-screening assessment (about 10 minutes, at application) → Technical interview → Face-to-face technical assessment at our Mohali office → HR and Leadership round → Background verification → Offer.

TERMS

  • Location: Work from our Mohali (Chandigarh Tricity) office.
  • Full time; immediate joiners or short notice periods preferred.
  • Background verification (BGV) is mandatory.
  • Compensation as per company policy and budget band.

How We Work

We value deep work, clarity, and execution over activity. You will be expected to clarify, plan, execute, and escalate with options, not wait to be told. Ownership sits with the individual, and outcomes are measured, not effort.

OUR VALUES

Agency – Be self-directed and proactive | Taste – Sweat the details and build with precision | Ownership – Take responsibility for outcomes | Mastery – Commit to continuous learning and growth | Impatience – Move fast and challenge inertia | Customer Obsession – Always put the customer first

WHY JOIN US

  • Build evaluation and testing infrastructure for enterprise-grade Agentic AI platforms.
  • Work on LLM evaluation, agent reliability, AI safety, and production-grade quality engineering.
  • Collaborate with AgentOps, engineering, security, and observability teams.
  • AI-first environment: work alongside the agentic platforms we build.
  • Great Place to Work-Certified, 14+ years, headquartered in Mohali.

Skills

PythonAnalyticsAiLlmLlmsRagTool CallingOpenaiAzure OpenaiLangchainLanggraphAi AgentAi AgentsAgenticAgentic AiAgentic WorkflowsMcpMlopsAmazon Bedrock

If a job posting appears fraudulent, asks for payment, contains misleading information, or violates our guidelines, please report it immediately. Our team will review it promptly, Jobaaj does not charge any fee from the applicants.

About Company

XenonStack is a cloud-native and data engineering company helping enterprises worldwide solve their digital transformation challenges. We design and deliver cloud-native solutions and data-driven platforms enabling enterprises to become adaptive, innovative and competitive.

Similar Jobs

View all
Loading...

Looking for something similar?

Bag Logo
Jobaaj
Get jobs like this by email

New Data Science Analytics & Machine Learning roles in Nagar.

Job Alert
Google hiring for Specific Roles Apply Now!
1 min ago
New Opportunity
Amazon is hiring freshers Apply Now!
5 min ago
Featured Jobs
Microsoft opening 50+ positions Apply Now!
10 min ago

One email a day. Unsubscribe any time.