Lead Consultant - QA(AI) Engineer

• We are looking for a hands-on Senior QA Consultant to lead end-to-end testing of AI and 
• Generative AI applications across enterprise environments. This role combines strong 
• expertise in traditional QA engineering with modern AI evaluation and validation 
• practices. The ideal candidate will drive quality assurance initiatives for RAG pipelines, 
• multi-agent systems, OCR and Speech-to-Text solutions while ensuring production
• grade quality outcomes in regulated industries such as Finance, Healthcare, and 
• Insurance. 
• The successful candidate will lead a small QA pod, collaborate closely with 
• engineering, AI/ML, product, and business teams, and act as the client-facing QA owner 
• for enterprise AI engagements. This role requires strong technical leadership, 
• automation expertise, AI evaluation capabilities, and excellent stakeholder 
• management skills. 
• Experience Requirements 

• 10–15 years of experience in Software QA / Test Engineering 
• 3–4 years of experience leading QA teams or delivery pods 
• Hands-on experience testing AI / GenAI applications in production environments 
• Experience working within regulated domains such as Finance, Healthcare, 

• Insurance, or similar industries 
• Key Responsibilities 
• Quality Engineering & Delivery 

• Own end-to-end quality assurance for AI/GenAI engagements including test strategy, 

planning, execution, defect management, reporting, and exit criteria 

• Lead and mentor QA pods of 3–6 engineers across automation frameworks, AI testing 

practices, and delivery execution 

• Design and execute functional, API, backend, integration, regression, performance, 

and AI evaluation test cases 

• Build and maintain scalable automation frameworks for Web, Mobile, and API testing 
• Drive continuous quality improvements through reusable accelerators, frameworks, 

datasets, and evaluation tools 

AI Application Testing & Evaluation 

• Define AI-specific test coverage including prompt validation, hallucination detection, 

groundedness checks, retrieval quality, tool-call correctness, latency, bias, and safety 

testing 

• Design and execute testing strategies for RAG architectures, multi-agent workflows, 

OCR systems, and Speech-to-Text applications 

• Build synthetic datasets covering edge cases, adversarial scenarios, persona-based 

workflows, and multimodal inputs 

• Implement LLM-as-a-Judge evaluation pipelines using customizable scoring rubrics 

and multi-model evaluations 

• Produce detailed AI quality scorecards and benchmarking reports across accuracy, 

relevancy, faithfulness, and safety metrics 

Automation, CI/CD & Tooling 

• Integrate AI evaluation and automation testing into Jenkins and Azure DevOps CI/CD 

pipelines 

• Build regression dashboards, quality gates, and automated reporting frameworks 
• Perform API testing, backend validation, database testing, and event-driven workflow 

validation 

• Conduct load and performance testing using modern performance engineering tools 
• Support cross-browser, parallel, and scalable automation execution environments 

Client & Stakeholder Management 

• Act as the client-facing QA lead for enterprise engagements 
• Conduct quality reviews, risk assessments, and stakeholder reporting sessions 
• Translate technical quality metrics into business-focused outcomes and 

recommendations 

• Collaborate with engineering, AI/ML, product, and leadership teams to ensure 

• successful delivery outcomes 
• Required Technical Skills - 
• Core QA & Automation 

• Functional Testing, Exploratory Testing, Requirements Traceability, and Test Design 

Techniques 

• API Testing using REST, GraphQL, Postman/Newman, RestAssured, and Requests 
• Backend and Database Testing across SQL Server, MySQL, Oracle, PostgreSQL, and 

ETL/Data Pipelines 

• Automation Frameworks using Selenium, Playwright, PyTest, and Appium 
• Performance Testing using JMeter, k6, or Locust 

Programming & Development - 

• Strong programming expertise in Python or Java 
• Knowledge of OOP concepts, design patterns, data structures, and unit testing 

frameworks 

• Ability to build utilities, custom assertions, CLI tools, and reusable automation 

components 

AI & Generative AI Testing 

• Hands-on experience testing AI/GenAI applications in enterprise environments 
• Knowledge of LLM platforms including OpenAI, Anthropic, Azure OpenAI, AWS 

Bedrock, and Vertex AI 

• Experience with orchestration frameworks such as LangChain, LlamaIndex, and 

Semantic Kernel 

• Understanding of multi-agent systems, memory handling, routing, tool calling, and 

guardrails 

• Expertise in RAG evaluation, vector databases, embeddings, retrieval quality, and 

hallucination testing 

• Familiarity with AI evaluation platforms such as RAGAS, DeepEval, Galileo, Arize 

Phoenix, Patronus, Okareo, or similar tools 

• Experience validating OCR systems and Speech-to-Text solutions using WER/CER and 

degradation testing methodologies 

Build, DevOps & Platforms 

• Azure DevOps, Jenkins, JIRA/Xray or Zephyr, GitHub/GitLab/Bitbucket 
• CI/CD pipelines, regression dashboards, and quality gate implementation 
• Exposure to Power Platform tools including Power Apps, Power BI, and Power 

Automate is a plus 

Domain & Compliance Expertise 

• Experience working in regulated industry environments such as Banking, Financial 

Services, Healthcare, Insurance, or Life Sciences 

• Understanding of compliance requirements including data privacy, PII/PHI handling, 

explainability, and AI governance practices 

Behavioral & Consulting Skills 

• Strong leadership and mentoring capabilities 
• Excellent verbal and written communication skills 
• Client-centric and solution-oriented mindset 
• Strong stakeholder management and collaboration abilities 
• Ownership mindset with proactive problem-solving approach 
• Ability to work effectively in fast-paced consulting environments 
• Continuous learner with strong interest in emerging AI testing methodologies and 

tools 

Preferred Qualifications 

• Bachelor’s degree in Computer Science, Engineering, or a related field 
• Master’s degree preferred 
• Certifications such as ISTQB Advanced, Certified Agile Tester, AWS/Azure AI 

Certifications, or equivalent 

• Experience with adversarial testing and responsible AI frameworks such as NIST AI 

RMF or ISO 42001 

• Exposure to MLOps / LLMOps observability and production monitoring 
• Contributions to QA or AI communities through blogs, speaking sessions, or open

source initiatives 

What Success Looks Like 

• High-quality delivery outcomes with measurable reduction in defect leakage 
• Successful implementation of reusable AI QA accelerators and automation 

frameworks 

• Strong stakeholder and client feedback across engagements 
• Growth and technical enablement of QA pod members under leadership 
• Continuous improvement of AI testing practices, evaluation frameworks, and delivery 

standards