QA Automation Engineer

QA Automation engineer

LLM Exposure Required | Remote | Full-Time

About the Role

We are looking for a passionate and detail-oriented QA Automation Engineer to join our growing QA & Engineering team. In this role, you will design, build, and maintain automated test frameworks for web, API, and AI-powered applications — with a special focus on testing Large Language Model (LLM)-based features. You will collaborate closely with quality analysts, product managers, and data analysts to ensure the quality, reliability, and performance of our software products.

Key Responsibilities

Automation & Testing

• Design, develop, and maintain automated test scripts for validating the output of an LLM-based product
• Integrate automated tests into CI/CD pipelines (GitHub Actions, Jenkins, etc.)
• Perform regression, smoke, sanity, and end-to-end testing across environments
• Identify, log, and track defects using tools like Jira
• Review test coverage and continuously improve test suite efficiency

LLM / AI Testing

• 
  Write automated and manual test cases for LLM-powered features (chatbots, copilots, summarization, etc.)
• Validate LLM outputs for accuracy, hallucination, tone, format, and edge-case behavior
• Design evaluation frameworks to measure prompt quality and model response consistency
• Collaborate with AI/ML teams to define acceptance criteria for model outputs
• Use AI coding assistants ( ChatGPT, Claude code) to accelerate test script development
• Stay updated with emerging LLM testing tools, techniques, and industry benchmarks

Collaboration & Reporting

• Work in Agile/Scrum sprints and participate in daily standups and sprint reviews
• Collaborate with cross-functional teams to understand requirements and define test strategies
• Produce clear test reports, coverage metrics, and quality dashboards
• Document test plans, test cases, and automation strategies

Required Skills & Qualifications

• 2–3 years of hands-on experience in test automation
• Proficiency in at least one language: Python, JavaScript, Java, or TypeScript
• Experience with one or more frameworks: Selenium, Playwright, Cypress, or Robot Framework
• Working knowledge of REST API testing (Postman, RestAssured, or similar)
• Familiarity with Git version control and branching strategies

  Looking to get Placed? Try our Placement Guarantee Plan

  Get Hired
• Exposure to CI/CD pipelines and DevOps practices
• Basic understanding of LLMs and how they work (GPT, Claude, Gemini, etc.)
• Ability to evaluate and test AI-generated outputs across key quality dimensions, including accuracy, bias, precision, recall, F1 score, and hallucinations
• Strong analytical, problem-solving, and communication skills

Good to Have (Preferred)

• Hands-on experience testing LLM-based products or AI features
• Familiarity with LLM APIs: OpenAI, Anthropic Claude, or Google Gemini
• Experience with prompt engineering and prompt evaluation techniques
• Understanding of ML concepts: tokenization, embeddings, RAG pipelines
• Hands-on experience with Google Sheets formulae

Perks & Benefits

• Work on cutting-edge AI & LLM products
• A remote-first culture with a lean, high-performing team of AI experts.
• Collaborative, inclusive, and supportive team culture
• Health & wellness benefits