Senior Data Bricks Engineer

Role - Senior Databricks Engineer

Experience Required: 8+ Years

Location: Hyderabad

Employment Type: Full Time, Permanent

Job Overview

We are seeking a Senior Databricks Engineer (8+ Years) to lead end-to-end enterprise data platform solutions across Azure and AWS environments. You will architect high-performance Lakehouse platforms, build scalable ETL/ELT pipelines using PySpark and Delta Lake, drive client/stakeholder engagements, and mentor technical teams.

Key Responsibilities

• Technical Leadership & Delivery: Lead the end-to-end design, development, and deployment of enterprise data platforms using Databricks, PySpark, SQL, and Delta Lake.
• Stakeholder Management: Conduct technical discussions, architecture reviews, effort estimations, and requirement-gathering workshops directly with clients and business stakeholders.
• Architecture & Optimization: Design scalable Lakehouse/Medallion architectures; optimize Spark applications, Databricks Workflows, and Job Clusters to minimize infrastructure costs and maximize speed.
• Governance & Security: Implement Unity Catalog for access control, metadata management, and data lineage while upholding ISO 27001 / ISMS compliance standards.
• Cloud & Pipeline Engineering: Build robust batch and real-time ETL pipelines integrating Databricks with Azure (ADF, Synapse) and/or AWS (S3, Glue, Lambda).
• DevOps & Best Practices: Drive automation via CI/CD pipelines (GitLab/GitHub), enforce coding standards, and mentor junior/mid-level data engineers.

Key Requirements

• 8+ years of total experience in Data Engineering & Big Data Analytics.
• 5+ years of hands-on expertise with Databricks, Apache Spark, PySpark, SQL, and Delta Lake.
• Deep understanding of Spark internals, cluster tuning, partitioning strategies, and Medallion Architecture.
• Proven hands-on experience with Azure (ADF/Synapse) or AWS (S3/Glue/Lambda).
• Solid experience with Unity Catalog, CI/CD DevOps practices, and data governance frameworks.
• Strong client-facing, communication, and technical team leadership capabilities.

• Education: Bachelors or Master’s degree in Computer Science, IT, or a related field.

Key Skills - Databricks PySpark Apache Spark Delta Lake Python SQL Azure Data Factory (ADF) AWS Glue Unity Catalog Medallion Architecture Data Engineering Lakehouse Architecture CI/CD ISO 27001