Test Automation Engineer – AI Solutions
An AI agent that applies to roles like this on employers' own sites. 25 applications free, no card needed.
Posted over 15 days ago, so the role may have closed. Confirm on the employer's page when you apply, or see similar jobs.
Job Description
Location: Bangalore / Mumbai
Role Overview
Seeking a QA Engineer with AI/ML testing experience to ensure the quality, reliability, and performance of AI-powered applications. The role involves validating AI models, LLM solutions, and data pipelines while identifying issues such as hallucinations, bias, model drift, and data quality concerns.
Key Responsibilities
- Test AI/ML models, LLM applications, RAG systems, chatbots, and data pipelines.
- Develop automated validation, regression, and performance testing frameworks.
- Evaluate model outputs for accuracy, consistency, relevance, bias, and hallucinations.
- Monitor production models for drift and performance degradation.
- Validate data quality and conduct fairness, bias, and explainability assessments.
- Bachelors/Masters degree in Computer Science, Engineering, or related field.
- 3 6 years of QA experience, including 1 2 years in AI/ML testing.
- Strong Python skills for automation and data analysis.
- Experience with RAGAS, DeepEval, Promptfoo, LangSmith, PyTest, Selenium, and Postman.
- Good understanding of the ML lifecycle, data quality validation, and pipeline testing.
- Strong analytical, communication, documentation, and stakeholder management skills.
- Prompt Engineering and LLM Red Teaming.
- MLOps tools: MLflow, SageMaker, Vertex AI.
- Vector Databases, Responsible AI, AI Safety, and Fairness Frameworks.
- A/B Testing, Shadow Deployments, CI/CD, and DevOps practices.
Skills
PythonData AnalysisAi/mlPrompt EngineeringAiMlIf a job posting appears fraudulent, asks for payment, contains misleading information, or violates our guidelines, please report it immediately. Our team will review it promptly, Jobaaj does not charge any fee from the applicants.
Similar Jobs
View allLooking for something similar?
New Data Science Analytics & Machine Learning roles in Bengaluru / Bangalore.
One email a day. Unsubscribe any time.

