Lead Data Engineer (10 Years With MS Fabric, Azure, ADF, Data Platforms Only)

Job Summary

We are looking for a Lead Data Engineer to help shape and grow our next-generation data engineering capability.

This is a hands-on technical leadership role for an experienced engineer who can set technical direction, establish engineering standards, mentor and elevate other engineers, and build enterprise-scale data solutions.

You will help design and evolve a modern, scalable, cloud-based data platform that may span technologies such as Microsoft Fabric, Databricks, Snowflake, and cloud-native services. Rather than being tied to a single technology stack, we are looking for someone who understands modern data architecture principles and can make thoughtful technology choices based on business, data, scalability, interoperability, and operational needs.

Just as importantly, we are looking for an AI-first engineering leader who can help the team rethink how we design, build, test, operate, and continuously improve data products using AI.

Key Responsibilities

Lead Engineering Excellence

• Provide hands-on technical leadership for a growing team of data engineers.
• Set engineering standards and reusable patterns across architecture, coding, testing, data modeling, documentation, observability, security, and deployment.
• Lead architecture discussions, technical design reviews, and code reviews.
• Mentor engineers and develop technical depth, ownership, problem-solving, and engineering judgment across the team.
• Translate complex initiatives into clear technical approaches and guide teams from design through production.
• Foster a culture of ownership, collaboration, curiosity, experimentation, continuous learning, and accountability.

Build & Evolve a Modern Data Platform

• Lead the architecture and engineering of scalable, secure, and governed enterprise data platforms.
• Design solutions across lakehouse, data warehouse, data lake, streaming, and data-product architectures based on the needs of the use case.
• Build batch, incremental, near-real-time, and event-driven data processing solutions.
• Create reusable engineering frameworks for ingestion, transformation, orchestration, data quality, metadata, and data delivery.
• Design scalable data models supporting operational, analytical, reporting, and AI use cases.
• Establish patterns that promote interoperability and reduce unnecessary platform and vendor dependencies.
• Evaluate technology choices based on performance, scalability, reliability, security, cost, maintainability, and business value.
• Remain hands-on and contribute directly to engineering solutions when needed.

Champion AI-First Engineering

• Champion the use of AI across the engineering lifecycle, including requirements, architecture, development, testing, documentation, troubleshooting, and operations.
• Establish responsible practices for reviewing, validating, securing, and governing AI-generated engineering artifacts.
• Promote spec-driven development, using well-defined requirements, data contracts, acceptance criteria, and automated tests.
• Design data platforms and products that are ready to support AI agents, RAG, semantic models, machine learning, and conversational analytics.
• Evaluate emerging AI and data technologies through focused experimentation and translate successful approaches into reusable engineering patterns.
• Help engineers use AI to increase productivity and innovation without compromising security, quality, or engineering accountability.

Build Reliable & Trusted Data Products

• Establish automated data quality, reconciliation, validation, and testing practices.
• Build observability across data pipelines, datasets, APIs, and platform services.
• Define operational expectations and reliability measures for critical data products.
• Lead root-cause analysis for significant issues and ensure corrective actions address systemic problems.
• Continuously improve platform reliability, resilience, performance, and operational efficiency.
• Treat data as a product with clear ownership, quality expectations, documentation, and consumer needs.

Design Enterprise Integration & Data Access

• Design secure, scalable integration patterns across enterprise applications, data platforms, and external data sources.
• Develop and guide the implementation of REST and GraphQL APIs.
• Design event-driven and serverless integration solutions.
• Establish standards for API lifecycle management, authentication, versioning, monitoring, and governance.
• Enable data access through appropriate patterns including APIs, data sharing, event streams, and governed analytical interfaces.
• Apply modern identity and access patterns such as OAuth 2.0, OpenID Connect, JWT, managed identities, service principals, or equivalent approaches.

Embed Governance, Security & Compliance

• Build data governance, lineage, metadata, data quality, security, privacy, and auditability into platform designs.
• Ensure engineering solutions align with applicable governance, risk, security, and compliance requirements.
• Apply appropriate engineering rigor to KYC, UCC, public-record, and other regulated or sensitive data domains.
• Partner with architecture, cybersecurity, governance, and compliance teams to address risks early in the engineering lifecycle.
• Maintain clear architectural decisions, data contracts, lineage, transformation logic, and operational documentation.

Partner & Influence Across the Organization

• Partner with product, architecture, analytics, AI, operations, and business teams to translate business opportunities into scalable technical solutions.
• Communicate architectural decisions, tradeoffs, risks, dependencies, and opportunities clearly to technical and non-technical audiences.
• Influence technical direction through expertise and collaboration rather than authority alone.
• Balance strategic platform modernization with near-term product and operational needs.
• Help establish a strong engineering community where ideas, patterns, and lessons are shared across teams.

Required Qualifications

• ·  Bachelor’s degree in Computer Science, Engineering, Information Systems, or related discipline, or equivalent practical experience.
• ·  10+ years of experience in data engineering, software engineering, data platforms, or related disciplines.
• ·  Demonstrated experience providing technical leadership, mentoring engineers, and leading complex initiatives from architecture through production.
• ·  Strong understanding of modern data platform and distributed data architecture principles.
• ·  Advanced SQL skills and strong expertise in data modeling, database design, and performance optimization.
• ·  Deep hands-on experience designing and building scalable ETL/ELT pipelines and reusable data engineering frameworks.
• ·  Significant experience with one or more modern cloud data platforms, such as Databricks, Microsoft Fabric, Snowflake, or comparable platforms.
• ·  Strong programming experience using Python, PySpark, Scala, Java, C#, or comparable languages.
• ·  Experience with cloud architecture and services across Azure, AWS, GCP, or comparable cloud environments.
• ·  Experience with APIs, distributed processing, event-driven architectures, and modern integration patterns.
• ·  Experience with automated testing, CI/CD, source control, observability, and modern DevOps practices.
• ·  Understanding of data governance, lineage, security, identity, and data quality practices.
• ·  Practical experience incorporating AI-assisted tools into engineering workflows.
• ·  Strong communication, technical judgment, problem-solving, and stakeholder-influence skills.
•  

Preferred Qualifications

• ·  Experience architecting solutions that span or integrate multiple data platforms or cloud technologies.
• ·  Experience with lakehouse architectures, Apache Spark, Delta Lake, Apache Iceberg, or other open data technologies.
• ·  Experience designing data products and applying data-as-a-product principles.
• ·  Experience with interoperable data-sharing patterns and open data architectures.
• ·  Experience building governed APIs or external data-delivery capabilities.
• ·  Experience supporting AI-oriented data architectures including semantic models, RAG, data agents, machine learning, or conversational analytics.
• ·  Experience with streaming and messaging technologies such as Kafka or comparable cloud-native services.
• ·  Experience with API management and enterprise integration architecture.
• ·  Familiarity with Infrastructure as Code and cloud automation.
• ·  Experience working with sensitive or highly governed data in financial services, legal, compliance, or regulatory environments.
• ·  Familiarity with KYC, UCC, public-record, GRC, or SOC 2 concepts.
• ·  Experience leading data platform modernization or migration initiatives.

Key Skills

Modern Data Platforms | Data Lakehouse & Warehousing | Databricks | Microsoft Fabric | Snowflake | Cloud Data Services | SQL | Python / PySpark | Apache Spark | Delta Lake / Apache Iceberg | Data Modeling | ETL/ELT | Data Products | REST & GraphQL APIs | Event-Driven Architecture | Data Quality & Observability | Data Governance & Lineage | CI/CD & DevOps | AI-Assisted Engineering | AI Agents & RAG

Our Interview Practices

To maintain a fair and genuine hiring process, we kindly ask that all candidates participate in interviews without the assistance of AI tools or external prompts. Our interview process is designed to assess your individual skills, experiences, and communication style. We value authenticity and want to ensure we’re getting to know you—not a digital assistant. To help maintain this integrity, we ask to remove virtual backgrounds and include in-person interviews in our hiring process. Please note that use of AI-generated responses or third-party support during interviews will be grounds for disqualification from the recruitment process.

Applicants may be required to appear onsite at a Wolters Kluwer office as part of the recruitment process.