Lead Data Quality Engineer – SQL / ETL / Microsoft Fabric

Company: Finarb Analytics Consulting Pvt. Ltd. Job Title: Lead Data Quality Engineer – SQL / ETL / Microsoft Fabric Location: Hyderabad, India Experience: 7–11 Years (Suggested) Employment Type: Full-Time Number of Openings: 1 About Finarb Finarb Analytics Consulting is a data, analytics, and technology consulting firm helping organizations transform complex data into actionable business insights through advanced analytics, business intelligence, artificial intelligence, and modern data platforms. Website: https://www.finarb.ai About the Role We are seeking an experienced, hands-on Lead Data Quality Engineer to drive data quality engineering, automated validation, reconciliation, and testing across enterprise data platforms. The ideal candidate will have extensive experience in advanced SQL, ETL/ELT testing, data reconciliation, data quality automation, and enterprise data warehousing, with exposure to Microsoft Azure and modern cloud data platforms. This role will be instrumental in ensuring data accuracy, completeness, consistency, and reliability during the organization's transition from SQL Server and Azure Data Factory to Microsoft Fabric. The successful candidate will lead data quality engineering activities, design scalable validation frameworks, establish testing standards, and mentor data quality engineers while remaining actively involved in complex technical investigations and automation development. The role requires a strong combination of technical expertise, quality engineering leadership, and the ability to collaborate with data architects, data engineers, Power BI developers, and business stakeholders. Key Responsibilities 1. Data Quality Engineering Leadership Lead end-to-end data quality engineering activities across enterprise data platforms. Define and implement data quality standards, testing methodologies, and validation strategies. Establish measurable data quality controls covering accuracy, completeness, consistency, uniqueness, validity, and integrity. Translate business requirements and data transformation specifications into comprehensive validation approaches. Review data quality test cases, reconciliation logic, and automation scripts developed by team members. Provide technical guidance and mentoring to data quality engineers. Drive continuous improvement in data quality processes, automation, and defect prevention. 2. Advanced SQL and Data Reconciliation Design and develop complex SQL-based reconciliation and validation solutions. Perform detailed source-to-target validation across multiple databases and data processing layers. Validate complex joins, aggregations, business rules, transformations, and derived calculations. Implement row-count, aggregate, field-level, and exception-based reconciliation. Develop reusable SQL validation procedures and automated reconciliation utilities. Investigate complex data discrepancies across large enterprise datasets. Optimize reconciliation queries and validation processes for performance and scalability. 3. ETL / ELT Testing and Validation Lead testing of ETL/ELT pipelines developed using SQL Server, Azure Data Factory, and Microsoft Fabric. Design comprehensive testing strategies covering data ingestion, transformation, integration, and downstream consumption. Validate full loads, incremental loads, CDC processes, upserts, and historical data processing. Develop test scenarios for data mapping, schema changes, null handling, duplicate detection, and exception processing. Lead regression testing following pipeline changes and production releases. Ensure appropriate validation coverage across staging, curated, and reporting layers. Collaborate with data engineering teams to identify and resolve transformation defects. 4. Microsoft Fabric Migration Quality Assurance Lead data quality and reconciliation activities during migration from SQL Server and Azure Data Factory to Microsoft Fabric. Define source-to-target validation strategies for Fabric Lakehouse, Warehouse, and data pipelines. Establish reconciliation criteria and acceptance thresholds for migrated datasets. Validate transformation logic and business calculations between legacy and Fabric environments. Oversee historical data reconciliation, incremental load testing, and parallel-run validation. Identify and investigate migration-related data discrepancies. Develop automated validation processes for large-scale data migration. Provide quality assessment and supporting evidence for migration readiness and production cutover. Support post-migration stabilization and production data quality monitoring. 5. Data Quality Automation and Framework Development Design and implement reusable, scalable data quality automation frameworks. Develop automated validation scripts using advanced SQL, Python, and other relevant technologies. Establish metadata-driven or configuration-driven reconciliation approaches where appropriate. Automate checks for completeness, uniqueness, referential integrity, data validity, and business rule compliance. Integrate data quality checks into ETL/ELT workflows and deployment pipelines. Implement automated exception reporting, quality dashboards, and defect identification. Reduce manual testing effort through standardized and repeatable validation processes. Evaluate and implement suitable data quality tools and frameworks. 6. Production Data Quality and Root Cause Analysis Lead investigation of complex production data quality incidents. Analyze discrepancies across source systems, transformation layers, and Power BI reporting outputs. Perform root cause analysis and recommend corrective and preventive actions. Identify recurring data quality issues and implement automated controls to prevent recurrence. Establish monitoring mechanisms and measurable data quality indicators. Coordinate with engineering and business teams to prioritize and resolve critical data defects. Ensure production data quality issues are appropriately documented, tracked, and resolved. 7. Data Quality Governance and Standards Establish standardized data validation and reconciliation methodologies. Define data quality metrics, thresholds, exception handling procedures, and reporting mechanisms. Maintain traceability between business rules, transformation logic, and validation test cases. Support data lineage, metadata management, and auditability requirements. Contribute to data quality documentation, test evidence, and release sign-off processes. Collaborate with data architects to ensure data quality controls are embedded within platform design. Promote a proactive, engineering-led approach to data quality. 8. Team Leadership and Cross-Functional Collaboration Provide technical leadership and mentoring to data quality engineers. Review validation designs, SQL scripts, automation frameworks, and testing deliverables. Coordinate testing priorities and activities across engineering teams. Work closely with the Principal Fabric Data Architect and Senior Fabric Data Engineer. Collaborate with Power BI developers to validate reporting data and business calculations. Communicate data quality risks, issues, and progress to technical and business stakeholders. Support technical planning, estimation, and delivery of data quality initiatives. Required Technical Skills Technology / Skill Expected Proficiency Advanced SQL / T-SQL Expert-level querying, reconciliation, optimization Data Quality Engineering Quality controls, validation strategy, defect prevention ETL / ELT Testing Advanced testing of complex data pipelines Data Reconciliation Source-to-target, record-level, aggregate and field-level Data Quality Automation Strong hands-on framework development Python Scripting, automation, data validation SQL Server Advanced development and troubleshooting Data Warehousing Dimensional modeling and transformation validation Azure Data Factory Pipeline understanding and testing Root Cause Analysis Complex production data investigations Test Strategy & Leadership Framework design, standards, technical mentoring Microsoft Fabric Working knowledge or hands-on exposure Required Qualifications Bachelor's degree in Computer Science, Information Technology, Engineering, or a related discipline. 7–11 years of professional experience in data quality engineering, ETL testing, data engineering, or data warehouse testing. Strong hands-on experience with advanced SQL and complex data reconciliation. Proven experience leading data quality or ETL testing activities for enterprise data platforms. Experience designing and implementing automated data validation frameworks. Strong knowledge of relational databases, data warehousing, and ETL/ELT architectures. Experience validating complex transformation logic and business calculations. Practical experience with SQL Server and Azure-based data environments. Experience performing root cause analysis of complex data discrepancies. Ability to independently design testing strategies and establish validation standards. Experience mentoring engineers and reviewing technical deliverables. Strong analytical, problem-solving, and stakeholder communication skills. Preferred / Good-to-Have Skills Hands-on experience with Microsoft Fabric, including Lakehouse, Warehouse, OneLake, and Fabric Data Factory. Experience with PySpark and Spark SQL for large-scale data validation. Familiarity with Great Expectations, Soda, dbt tests, or equivalent data quality frameworks. Experience designing metadata-driven reconciliation frameworks. Knowledge of Delta Lake and medallion architecture. Experience leading data validation for cloud migration or data platform modernization projects. Familiarity with Power BI semantic models and reporting validation. Experience with Azure DevOps, Git, and CI/CD pipelines. Exposure to Microsoft Purview, data lineage, and metadata management. Experience with data observability, quality scorecards, and automated anomaly detection. Familiarity with data governance, security, and audit requirements. Key Competencies Strong technical leadership with a hands-on engineering mindset. Excellent analytical and complex problem-solving skills. Ability to design scalable validation strategies and frameworks. Strong ownership of data quality outcomes. Attention to detail and commitment to data accuracy. Ability to mentor engineers and establish engineering standards. Effective collaboration across data engineering, architecture, and BI teams. Clear communication of data quality risks and technical findings. Ability to manage competing priorities during migration and production operations. What Success Looks Like Establish a comprehensive and scalable data quality engineering framework. Lead reliable source-to-target reconciliation across legacy and Microsoft Fabric environments. Automate critical data validation processes and reduce manual testing effort. Ensure data accuracy and consistency throughout migration activities. Detect and resolve critical data discrepancies before they affect downstream reporting. Establish standardized quality metrics, testing practices, and validation controls. Mentor data quality engineers and improve overall team capabilities. Provide reliable validation evidence for migration cutover and production releases. Reduce recurring production data quality defects through proactive controls. Build a sustainable data quality engineering capability across the organization. Why Join Finarb? Take ownership of data quality engineering within enterprise-scale data modernization initiatives. Work on Microsoft Fabric, Azure, and advanced data engineering platforms. Lead the design and implementation of scalable data quality automation frameworks. Collaborate with experienced data architects, engineers, and analytics professionals. Influence technical standards and best practices across data engineering teams. Work on complex data migration and transformation programs. Combine hands-on engineering responsibilities with technical leadership opportunities. Job Location: Hyderabad | Employment: Full-Time | Experience: 7–11 Years | Open Positions: 1