Gen AI Test Engineer

Department Icon Data Science Analytics & Machine Learning
Posted: 7 months ago
4-7 years
Pune, Maharashtra
Data Science Analytics & Machine Learning
work from office
Full Time

Posted: 7 months ago
This job's closing date (29 May 2026) has passed, so the role may already be filled. See live similar jobs.
Let Jobaaj One apply for you

An AI agent that applies to roles like this on employers' own sites. 25 applications free, no card needed.

Job Description
About Company
Similar Jobs
Please verify your account first! Send OTP

Job Description

Role: QA Engineer Generative AI / Responsible AI Testing

Experience: 47 Years

Location: Open to any Orion Innovation Office in India ( Chennai/Pune/ Mumbai/Kochi/Coimbatore/Hyderabad)

Employment Type: FTE

Role Overview

We are seeking a detail-oriented QA Engineer with experience in Generative AI and Responsible AI testing to ensure the quality, safety, and reliability of AI-powered applications, including LLM-driven chatbots and intelligent systems. The role will focus on functional, adversarial, and ethical validation of GenAI solutions while supporting automation and performance testing initiatives.

Key Responsibilities

GenAI & Responsible AI Testing

  • Design and execute prompt-based test cases covering functionality, bias, safety, and edge scenarios.
  • Craft adversarial prompts to stress-test LLM behavior and identify vulnerabilities.
  • Validate hallucination resistance, output consistency, and prompt reliability.
  • Evaluate AI systems for bias, fairness, transparency, and ethical compliance.
  • Test content moderation and safety filters for GenAI applications.
  • Perform end-to-end QA for GenAI chatbot conversational flows.

Functional & System Testing

  • Design test scenarios, test cases, and regression suites.
  • Conduct System Integration Testing (SIT), Smoke Testing, Regression Testing, and Bug Testing.
  • Execute manual and automated test cycles across releases.
  • Maintain defect logs, triage defects, and track resolution.

Automation & Performance Testing

  • Develop and maintain automation scripts using Python-based frameworks.
  • Build automated tests using Pytest / Selenium / Robot Framework.
  • Support load and performance testing to assess system scalability.

BI & Data Validation Testing

  • Validate BI dashboards, reports, and visualizations.
  • Ensure data accuracy, reporting reliability, and performance across tools like Tableau and Power BI.
  • Support data storytelling and analytics validation.

Agile & DevOps Collaboration

  • Participate in Sprint Planning, Backlog Grooming, and Daily Stand-ups.
  • Collaborate with Developers, Product Owners, and Analysts to streamline QA processes.
  • Support build deployments across QA, UAT, and Production environments.
  • Provide feature demos and stakeholder updates.

Required Skills & Competencies

Testing Expertise

  • Software Testing Life Cycle (STLC) & SDLC
  • Test case design & execution
  • Defect management & triaging
  • Regression & integration testing

GenAI / AI Testing

  • LLM testing & validation
  • Prompt engineering for testing
  • Responsible AI principles
  • Bias & fairness evaluation

Automation & Tools

  • Python
  • Pytest
  • Selenium / Robot Framework
  • LangChain (preferred)

BI & Data Tools

  • Tableau
  • Power BI
  • Advanced Excel

Project & QA Tools

  • Jira
  • Confluence
  • Azure DevOps

Qualifications

  • Bachelors Degree in Computer Science, Engineering, or related field.
  • ISTQB CTFL certification preferred.

Nice to Have

  • Knowledge of Machine Learning / Deep Learning concepts.
  • Experience with GenAI governance frameworks.
  • Exposure to statistical analysis and data science fundamentals.

Soft Skills

  • Strong analytical & problem-solving skills
  • Excellent communication & stakeholder management
  • Attention to detail
  • Ability to work in Agile teams

Skills

Data ValidationPythonData ScienceDeep LearningMachine LearningPrompt EngineeringAnalyticsAi

If a job posting appears fraudulent, asks for payment, contains misleading information, or violates our guidelines, please report it immediately. Our team will review it promptly, Jobaaj does not charge any fee from the applicants.

About Company

Orion Innovation is a global digital transformation partner that helps Fortune 500 companies build and launch innovative digital products and services. We work with leading brands across many industries, and our services include strategy, design, engineering, and operations.

Similar Jobs

View all
Loading...

Looking for something similar?

Bag Logo
Jobaaj
Get jobs like this by email

New Data Science Analytics & Machine Learning roles in Pune.

Job Alert
Google hiring for Specific Roles Apply Now!
1 min ago
New Opportunity
Amazon is hiring freshers Apply Now!
5 min ago
Featured Jobs
Microsoft opening 50+ positions Apply Now!
10 min ago

One email a day. Unsubscribe any time.