Quality Assurance Engineer

Department Icon Data Science Analytics & Machine Learning
Posted: 3 months ago
5-7 years
Delhi, Delhi
Data Science Analytics & Machine Learning
work from office
Full Time

Posted: 3 months ago
This job's closing date (21 Sep 2026) has passed, so the role may already be filled. See live similar jobs.
Let Jobaaj One apply for you

An AI agent that applies to roles like this on employers' own sites. 25 applications free, no card needed.

Job Description
About Company
Similar Jobs
Please verify your account first! Send OTP

Job Description

Role at a Glance

Title: QA Automation Engineer - AI

Team: Engineering

Employment Type: Full-time

Location: Remote (APAC)

Why This Role Matters

At Hupo, AI quality is the product - a wrong score, hallucinated feedback, or a broken voice turn is a customer-facing defect, and in a regulated BFSI context, a compliance risk. This role owns and scales the testing of Hupos AI pipelines, working closely with QA, Engineering, and Product as the platform grows.

Its a hands-on, high-ownership role in a fast-moving environment - youll be a key part of deciding, with evidence, whether each release is safe to ship.

What Youll Do

  • Build and maintain automated testing for AI voice and chat agents, from single conversational turns to full roleplay flows.
  • Test LLM output quality - correctness, consistency, structured output, language fidelity, prompt regression - using evaluation harnesses (LLM-as-judge, golden datasets, tolerance-based assertions for non-determinism).
  • Validate RAG pipelines: retrieval relevance, grounding/faithfulness, and answer quality.
  • Test voice pipelines: STT/TTS accuracy and real-time, low-latency behavior across languages.
  • Automate across UI/E2E, API, and AI output layers, and build CI/CD from scratch - lint, type-check, tests, evals, coverage gates, deployment checks.
  • Own release quality: regression strategy, catching breakages early, and clear go/no-go calls.
  • Cover database and load/performance testing; extend automation to desktop and mobile clients.
  • Take over and extend existing QA automation, and strengthen shared frameworks, tooling, and QA processes/standards.

What Success Looks Like

Within your first few months, youll:
  • Ramp on Hupos existing QA automation suites and take clear ownership of them.
  • Extend test coverage for LLM and voice-agent behavior, including new evaluation harnesses for non-deterministic output.
  • Contribute CI/CD improvements - coverage gates, deployment checks - that catch regressions before they reach customers.

Youll also be expected to continuously:
  • Adapt and expand impact as priorities shift
  • Take ownership of new problems as they emerge

What Were Looking For

Must-Haves

  • 5+ years as a QA Automation Engineer, with a proven track record testing AI systems - not just traditional software.
  • Builder mindset: established testing frameworks, standards, and CI/CD-integrated automation from scratch.
  • Hands-on experience testing conversational AI - voice and/or chat agents - on real, shipped projects.
  • Strong grasp of LLMs, prompt engineering, and RAG - designing tests for nondeterministic output and evaluating retrieval/generation quality.
  • Understanding of voice pipelines (speech-to-text, text-to-speech) and how to test them in automation.
  • Hands-on with LLM evaluation/observability tooling - e.g. Langfuse, LangSmith, DeepEval, RAGAS.
  • Proficient in UI/E2E (e.g. Playwright) and API test automation; experience building CI/CD pipelines from scratch.
  • Comfort with database testing, load/performance testing, and desktop/mobile test automation.
  • Solid Git discipline, basic cloud knowledge, and familiarity with the agile sprint lifecycle.
  • Track record in fast-paced, fast-shipping environments - ramps quickly on unfamiliar systems, pragmatic about process.

Nice-to-Haves

  • TypeScript - for reading the codebase and contributing test/automation code directly.
  • Voice & real-time: LiveKit / WebRTC; STT/TTS via Azure Speech, Google Cloud Speech, Deepgram, or self-hosted Whisper.
  • AI/ML infra & eval tooling: GPU inference (vLLM / TGI / Triton), Azure OpenAI, golden datasets, HITL pipelines, Braintrust, Promptfoo.
  • QA & cloud tooling: Cypress, Postman, multi-cloud (AWS/Azure/GCP), Infrastructure-asCode (Terraform/Pulumi/CloudFormation), Docker/Kubernetes.
  • Also useful: product analytics familiarity (e.g. PostHog), multilingual/localized testing experience, and production monitoring/error-tracking tools.

Skills

Ai/mlPrompt EngineeringAnalyticsAiGoogle CloudMlQuality Assurance

If a job posting appears fraudulent, asks for payment, contains misleading information, or violates our guidelines, please report it immediately. Our team will review it promptly, Jobaaj does not charge any fee from the applicants.

About Company

We are an AI-native start-up building sales enablement products in the banking, financial services, and insurance industry; already trusted by dozens of enterprise customers (including Fortune 500 companies).

Similar Jobs

View all
Loading...

Looking for something similar?

Bag Logo
Jobaaj
Get jobs like this by email

New Data Science Analytics & Machine Learning roles in Delhi.

Job Alert
Google hiring for Specific Roles Apply Now!
1 min ago
New Opportunity
Amazon is hiring freshers Apply Now!
5 min ago
Featured Jobs
Microsoft opening 50+ positions Apply Now!
10 min ago

One email a day. Unsubscribe any time.