An AI agent that applies to roles like this on employers' own sites. 25 applications free, no card needed.
Posted over 15 days ago, so the role may have closed. Confirm on the employer's page when you apply, or see similar jobs.
Job Description
About Us:
ALOIS Solutions is a leading Talent and Technology Solutions company, focused on helping businesses across the country build stronger teams, smarter processes, and sustainable growth.
We work with organisations in diverse industries to deliver integrated workforce and technology solutions, from innovative hiring strategies to end-to-end digital transformation. Our approach combines data, insight, and human expertise to help clients solve complex workforce challenges with clarity and confidence.
Through our deep industry experience and commitment to excellence, we provide scalable, high-performance solutions that deliver measurable impact. Whether its onshore project delivery, workforce optimisation, or specialised technology services, ALOIS Australia ensures results that truly make a difference.
Job Title – Senior AI Quality Assurance (QA) Engineer
Job Location – Bengaluru (Hybrid)
Experience – 10+ Years
Employment Type – Permanent / Full-Time
Availability to Join : Immediate to 2 Weeks.
About the Role
We are looking for an experienced Senior AI Quality Assurance (QA) Engineer to ensure the quality, reliability, and production readiness of our enterprise-grade AI platform powering Natural Language-to-SQL (NL2SQL) capabilities.
This role extends beyond traditional software testing and focuses on the evaluation and quality engineering of Large Language Model (LLM) applications. You will own AI benchmarking, automated evaluations, observability, prompt regression testing, and end-to-end validation of AI workflows. Working closely with AI Engineers, Data Engineers, and Product teams.
Required Skills:
- Python & Pytest Automation
- LLM / Generative AI / Conversational AI Testing
- AI Benchmarking & Prompt Regression Testing
- LangSmith / MLflow / Langfuse / Arize Phoenix
- SQL & REST API Testing
- Strong debugging and root cause analysis
- Experience with AI evaluation metrics (Precision, Recall, F1, EM, MRR, Latency)
Good to Have:
- RAG, LangGraph, OpenAI, Azure OpenAI
- Vector Databases (Pinecone, Milvus, Weaviate)
- Playwright
- BFSI Domain Experience
Skills
PythonQuality AssuranceRoot Cause AnalysisLarge Language ModelAiSqlIf a job posting appears fraudulent, asks for payment, contains misleading information, or violates our guidelines, please report it immediately. Our team will review it promptly, Jobaaj does not charge any fee from the applicants.
Similar Jobs
View all
New Data Science Analytics & Machine Learning roles in Bengaluru / Bangalore.
One email a day. Unsubscribe any time.
