Data Engineer

Lead SDET Data Engineering / Data QA (Databricks & PySpark)

Location: Indore / Pune

Experience: 1011 Years (Relevant 57 years hands-on)

Compensation: Highly Cometitive

Employment Type: Full-Time

Joining: Immediate / Serving Notice Period Candidates Preferred

Role Overview

We are looking for a hands-on Lead SDET / Data QA Engineer with strong expertise in data validation and data pipeline testing within modern data platforms. This role is execution-focused, similar to development work, and is ideal for professionals who enjoy validating complex data transformations built by data engineers.

The role is positioned as a Tester organizationally but requires deep understanding of data engineering concepts, SQL, and PySpark-based pipelines. This is not a managerial role; however, slight team lead exposure is preferred for future scaling.

Key Responsibilities

Data Validation & Pipeline Testing

• Validate large-scale data pipelines built using Azure Data Factory and Databricks
• Perform deep data validation across multiple layers (raw curated consumption)
• Validate data transformations, joins, aggregations, and business logic implemented by data
• engineers
• Perform SQL-based reconciliation, data completeness, and accuracy checks
• Review data pipeline logic and collaborate with data engineers to identify defects early

Execution-Focused Testing

• Write and execute PySpark / Python-based test logic for validating transformations
• Perform heavy-weight testing comparable to development work (not basic QA checks)
• Debug pipeline failures and analyze root causes at data and code level
• Ensure data correctness across batch pipelines (streaming optional, not mandatory)

CI/CD & Platform Collaboration

• Integrate testing workflows into Azure DevOps CI/CD pipelines
• Validate data outputs post-deployment across environments
• Work closely with data engineers, platform teams, and stakeholders to ensure release quality

Leadership (Lightweight)

• Provide technical guidance to junior testers (if assigned)
• Review test logic and validation queries
• Help standardize data QA best practices (process can be trained, coding cannot)

Mandatory Skills & Technologies

Data Engineering Testing Experience: 57 years (hands-on)

SQL: Strong (joins, aggregations, reconciliation, validation queries)

Databricks: Hands-on experience

PySpark / Python: Strong working knowledge (writing validation logic)

Azure Data Factory (ADF): Pipeline understanding and validation

Azure DevOps: CI/CD pipeline integration

Strong understanding of data engineering concepts and pipeline architecture