Data Engineer

Lead SDET Data Engineering / Data QA (Databricks & PySpark)

Location: Indore / Pune

Experience: 1011 Years (Relevant 57 years hands-on)

Compensation: Highly Cometitive

Employment Type: Full-Time

Joining: Immediate / Serving Notice Period Candidates Preferred

Role Overview

We are looking for a hands-on Lead SDET / Data QA Engineer with strong expertise in data validation and data pipeline testing within modern data platforms. This role is execution-focused, similar to development work, and is ideal for professionals who enjoy validating complex data transformations built by data engineers.

The role is positioned as a Tester organizationally but requires deep understanding of data engineering concepts, SQL, and PySpark-based pipelines. This is not a managerial role; however, slight team lead exposure is preferred for future scaling.

Key Responsibilities

Data Validation & Pipeline Testing

Validate large-scale data pipelines built using Azure Data Factory and Databricks

Perform deep data validation across multiple layers (raw curated consumption)

Validate data transformations, joins, aggregations, and business logic implemented by data

engineers

Perform SQL-based reconciliation, data completeness, and accuracy checks

Review data pipeline logic and collaborate with data engineers to identify defects early

Execution-Focused Testing

Write and execute PySpark / Python-based test logic for validating transformations

Perform heavy-weight testing comparable to development work (not basic QA checks)

Debug pipeline failures and analyze root causes at data and code level

Ensure data correctness across batch pipelines (streaming optional, not mandatory)

CI/CD & Platform Collaboration

Integrate testing workflows into Azure DevOps CI/CD pipelines

Validate data outputs post-deployment across environments

Work closely with data engineers, platform teams, and stakeholders to ensure release quality

Leadership (Lightweight)

Provide technical guidance to junior testers (if assigned)

Review test logic and validation queries

Help standardize data QA best practices (process can be trained, coding cannot)

Mandatory Skills & Technologies

Data Engineering Testing Experience: 57 years (hands-on)

SQL: Strong (joins, aggregations, reconciliation, validation queries)

Databricks: Hands-on experience

PySpark / Python: Strong working knowledge (writing validation logic)

Azure Data Factory (ADF): Pipeline understanding and validation

Azure DevOps: CI/CD pipeline integration

Strong understanding of data engineering concepts and pipeline architecture