Data Engineer

Job Summary:

We are seeking a highly skilled Data Engineer to lead the development of our enterprise-grade Data Lakehouse platform on Databricks. This role is central to building scalable, reliable, and high-performance data pipelines that transform diverse data types including structured, semi-structured, and unstructured into curated, analytics-ready assets.

Experience:

• 3- 10 years of hands-on experience in data engineering, with a focus on building scalable data pipelines.
• Proven experience building and maintaining data platforms on a major cloud provider (Azure preferred).
• Hands-on expertise with the Databricks platform, including PySpark, Delta Lake, and performance tuning.
• Practical experience with modern data transformation tools, preferably dbt.
• Strong understanding of data modeling concepts and architectural patterns (e.g., Kimball, Inmon, Data Vault).
• Exposure to CI/CD for data pipelines and infrastructure-as-code principles.
• Familiarity with data governance principles and tools like
  
  Unity Catalog.
• Experience connecting data platforms to BI tools like Power BI is a plus.

Key Responsibilities:

• Architect and implement robust ELT pipelines to ingest data from a wide range of sources, including databases, APIs, files, and event streams, with support for schema evolution and data validation.
• Design and manage end-to-end data workflows using the Medallion Architecture (Bronze, Silver, Gold) to incrementally refine and structure data for downstream
• Develop scalable data transformation jobs using PySpark and SQL, leveraging frameworks like dbt to build modular, testable, and well-documented data models.
• Apply data governance and quality controls using Databricks Unity Catalog and Delta Live Tables (DLT), ensuring metadata management, lineage tracking, and secure access to sensitive data.

• Looking to get Placed? Try our Placement Guarantee Plan

  Get Hired
  Optimize performance and cost-efficiency of Databricks jobs through effective partitioning, Z-Ordering, caching, and cluster tuning.
• Integrate the Lakehouse with downstream analytics and AI systems, exposing clean data views and APIs for BI dashboards and machine learning workflows.
• Implement CI/CD practices for data pipelines, collaborating with DevOps teams to automate testing, deployment, and monitoring using tools like Azure DevOps or GitHub Actions.
• Ensure compliance with data privacy and security standards, including HIPAA and other relevant regulations.
• Communicate technical architecture and design decisions effectively to both technical and non-technical stakeholders.
• Evaluate and adopt emerging tools and best practices within the Databricks ecosystem to continuously improve platform capabilities.
• Contribute to internal data engineering libraries, reusable pipeline templates, and documentation to support team scalability.

Work in an agile environment following Scrum or Kanban methodologies