Senior Databricks Data Engineer

YOUR ROLE

• Design and deliver enterprise-scale data solutions using Databricks Lakehouse architecture and Delta Lake.
• Lead the development of scalable batch and real-time data pipelines using PySpark, Spark SQL, and Structured Streaming.
• Implement and govern data security, lineage, access management, and compliance standards using Unity Catalog.
• Drive platform performance optimization, scalability improvements, and cost management initiatives.
• Establish and enhance CI/CD pipelines, DevOps practices, automated testing frameworks, and Infrastructure-as-Code implementations.
• Provide technical leadership through code reviews, mentoring, and engineering best practices.
• Partner with architects, business stakeholders, and delivery teams to translate business requirements into scalable data solutions.
• Troubleshoot and resolve complex production issues while ensuring platform reliability and operational excellence.
• Support solution architecture, technical estimations, framework development, and project delivery activities.

YOUR PROFILE

• 6-8 years of experience in Data Engineering, including significant hands-on experience with Databricks.Advanced expertise in Python, PySpark, Spark SQL, and large-scale data processing frameworks.
• Strong experience with Delta Lake, Unity Catalog, Databricks Workflows, and Delta Live Tables.
• Proven experience designing and implementing cloud-based data platforms on Azure, AWS, or GCP.In-depth understanding of data modeling, data warehousing, and modern Lakehouse architectures.
• Experience working with real-time data technologies such as Kafka, Event Hubs, and Structured Streaming.Strong knowledge of Git, DevOps practices, CI/CD pipelines, and Infrastructure-as-Code methodologies.
• Demonstrated ability to lead technical initiatives, mentor team members, and collaborate with global stakeholders.
• Exposure to MLflow, Microsoft Fabric, Azure Synapse, Snowflake, and enterprise data governance frameworks is desirable.
• Bachelor's or Master's degree in Computer Science, Information Technology, or a related discipline.