AI Test Engineer

Position Overview:

• Experience in software testing, specializing in Artificial Intelligence (AI) systems and assure quality of an Agentic AI platform built using LLMs, LangChain/LangGraph, CrewAI, and related orchestration frameworks. The role includes testing autonomous agent behaviour, multi agent workflows, RAG pipelines, tool integrations, and domain-specific flows across insurance scenarios such as claims, underwriting, and policy servicing. The role involves validating AI/ML models, ensuring robustness of end-to-end machine learning workflows, and driving quality engineering practices across data, models, and deployment environments. The engineer will collaborate with data scientists, ML engineers, and DevOps teams to ensure trustworthy, scalable, and production-ready AI solutions.

• strong analytical skills, experience with AI workflow testing, and the ability to design robust test suites for dynamic reasoning systems.
• Strong understanding of both manual and automated testing methodologies, with the ability to design, develop, and execute test scripts.
• Responsible for the testing and delivery of parts of a product, in accordance with the customers requirements and interpersonal quality norms.
• Strong analytical and problem-solving skills, along with a proactive approach to identify and resolve issues, are essential.
• Responsible for ensuring that the testing process is efficient, effective, and aligned with project goals and timelines.

Responsibilities /

• Design and execute test strategies for machine learning models (classification, regression, NLP, CV, etc.).
• Validate model accuracy, fairness, bias detection, explainability, and robustness under varying datasets.
• Perform adversarial testing and edge-case validation for AI systems.
• Test autonomous agents, multi-step reasoning paths, and state transitions.
• Validate correctness, grounding, consistency, and safety of LLM outputs.
• Evaluate prompt robustness and behavioral variations across scenarios.
• Validate retrieval accuracy, grounding quality, and hallucination reduction.
• Test vector store behavior, document chunking logic, and retriever configurations.
• Validate fallback behaviors when tools or external services fail.
• Execute adversarial, prompt-injection, and red team style testing.
• Validate compliance with data privacy, business rules, and insurance guidelines.
• Benchmark LLM latency, throughput, and multi-agent performance.
• Validate concurrency handling, degradation logic, and retry mechanisms.
• Document test cases, behavior maps, prompt variations, and evaluation reports.
• Work closely with AI engineers and SMEs to refine agent workflows.
• Functional & Non-Functional Testing:

• o Conduct API testing for AI services and model endpoints. o Validate performance, scalability, and latency of AI inference services.
• o Perform security testing for data pipelines and model-serving endpoints. •
• Quality Governance & Compliance:
• o Ensure compliance with AI ethics, data privacy, and regulatory standards.
• o Establish KPIs for model quality (precision, recall, F1-score, drift detection).
• o Document test strategies, results, and provide audit-ready evidence.
• • Collaboration & Leadership:
• o Work closely with data scientists, ML engineers, and DevOps teams to integrate testing into the AI lifecycle.
• o Mentor junior testers and contribute to building reusable test accelerators.
• o Represent the testing function in client discussions, providing insights on AI quality assurance.

• Develop automated test frameworks for AI/ML pipelines.
• Implement synthetic data generation for edge-case testing.
• Conduct bias and fairness testing across diverse datasets.
• Validate monitoring dashboards for model drift and performance degradation.
• Perform regression testing for retrained models.
• Create reusable test assets for AI projects (scripts, datasets, frameworks).
• Provide detailed defect analysis and SLA-driven closure.

Required Skills

• Strong experience in software testing (manual + automation) with focus on AI systems.
• Hands-on expertise in Python for test automation, data validation, and AI testing scripts.
• Knowledge of Gen AI frameworks (Langchain, Langgraph, CrewAI).
• Familiarity with CI/CD pipelines (Jenkins, GitHub Actions, GitLab CI).
• Strong understanding of data validation, model evaluation metrics, and drift detection.
• API testing experience (REST, GraphQL) with tools like Postman or REST Assured.
• Solid knowledge of cloud platforms (AWS, Azure, GCP) for AI deployments.
• Exposure to AI ethics and compliance testing.
• Knowledge of containerization & orchestration (Docker, Kubernetes).
• Experience with Big Data ecosystems (Spark, Hadoop) for large-scale AI testing.
• Familiarity with observability tools (Prometheus, Grafana, ELK stack).
• Understanding of security testing for AI pipelines (data poisoning, adversarial attacks).