Gen AI Test Engineer

Role: QA Engineer Generative AI / Responsible AI Testing

Experience: 47 Years

Location: Open to any Orion Innovation Office in India ( Chennai/Pune/ Mumbai/Kochi/Coimbatore/Hyderabad)

Employment Type: FTE

Role Overview

We are seeking a detail-oriented QA Engineer with experience in Generative AI and Responsible AI testing to ensure the quality, safety, and reliability of AI-powered applications, including LLM-driven chatbots and intelligent systems. The role will focus on functional, adversarial, and ethical validation of GenAI solutions while supporting automation and performance testing initiatives.

Key Responsibilities

GenAI & Responsible AI Testing

• Design and execute prompt-based test cases covering functionality, bias, safety, and edge scenarios.
• Craft adversarial prompts to stress-test LLM behavior and identify vulnerabilities.
• Validate hallucination resistance, output consistency, and prompt reliability.
• Evaluate AI systems for bias, fairness, transparency, and ethical compliance.
• Test content moderation and safety filters for GenAI applications.
• Perform end-to-end QA for GenAI chatbot conversational flows.

Functional & System Testing

• Design test scenarios, test cases, and regression suites.
• Conduct System Integration Testing (SIT), Smoke Testing, Regression Testing, and Bug Testing.
• Execute manual and automated test cycles across releases.
• 
  Maintain defect logs, triage defects, and track resolution.

Automation & Performance Testing

• Develop and maintain automation scripts using Python-based frameworks.
• Build automated tests using Pytest / Selenium / Robot Framework.
• Support load and performance testing to assess system scalability.

BI & Data Validation Testing

• Validate BI dashboards, reports, and visualizations.
• Ensure data accuracy, reporting reliability, and performance across tools like Tableau and Power BI.
• Support data storytelling and analytics validation.

Agile & DevOps Collaboration

• Participate in Sprint Planning, Backlog Grooming, and Daily Stand-ups.
• Collaborate with Developers, Product Owners, and Analysts to streamline QA processes.
• Support build deployments across QA, UAT, and Production environments.
• Provide feature demos and stakeholder updates.

Required Skills & Competencies

Testing Expertise

• Software Testing Life Cycle (STLC) & SDLC
• Test case design & execution
• Defect management & triaging
• Regression & integration testing

• LLM testing & validation
• Prompt engineering for testing
• Responsible AI principles
• Bias & fairness evaluation

Automation & Tools

• Python
• Pytest
• Selenium / Robot Framework
• LangChain (preferred)

BI & Data Tools

• Tableau
• Power BI
• Advanced Excel

Project & QA Tools

• Jira
• Confluence
• Azure DevOps

Qualifications

• Bachelors Degree in Computer Science, Engineering, or related field.
• ISTQB CTFL certification preferred.

Nice to Have

• Knowledge of Machine Learning / Deep Learning concepts.
• Experience with GenAI governance frameworks.
• Exposure to statistical analysis and data science fundamentals.

Soft Skills

• Strong analytical & problem-solving skills
• Excellent communication & stakeholder management
• Attention to detail
• Ability to work in Agile teams