Senior Databricks Engineer

Key Responsibilities

Design &

• Development:
• Build and maintain scalable ETL/ELT data pipelines using Apache Spark and Databricks.
• Develop Delta Lake architectures and optimize data storage for performance and cost.
• Automate data ingestion, transformation, and orchestration processes across multiple data sources.
• Build different stages of medallion architecture such as Bronze, Silver and Gold layer

Data Architecture & Modeling:

• Collaborate with data architects to design data lakehouse and warehouse solutions.
• Implement data quality frameworks, versioning, and governance standards (e.g., Unity Catalog).
• Optimize Spark jobs, cluster configurations, and query performance.
• Cloud & Infrastructure:
  
• Work with Azure Databricks or AWS to configure and manage cloud-based data platforms.
• Develop and manage CI/CD pipelines for Databricks notebooks, workflows, and jobs.
• Implement security best practices for data access, encryption, and compliance.

Collaboration & Leadership:

• Partner with data scientists, analysts, and product teams to enable data-driven decisions.
• Provide technical mentorship to junior engineers.
• Contribute to strategic planning and architecture reviews.

Required Qualifications

• Bachelors or Masters degree in Computer Science, Information Systems, Engineering, or related field.
• 6+ years of experience in data engineering or software development.
• 4+ years hands-on experience with Databricks and Apache Spark.
• Strong proficiency in Python, SQL, and PySpark.
• Experience with Delta Lake, Data Lakehouse, and Data Warehousing concepts.
• Solid understanding of data modeling, ETL design, medallion architecture and distributed data processing.
• Familiarity with CI/CD tools (Azure DevOps, GitHub Actions, Jenkins, etc.).
• Cloud platform expertise in Azure, AWS, or GCP.

Preferred Qualifications

• Experience with machine learning pipelines in Databricks (MLflow).
• Knowledge of data governance and compliance frameworks.
• Experience integrating Databricks with BI tools like Power BI, Tableau, or Looker.
• Certifications such as:
• Databricks Certified Data Engineer Associate or Professional
• Azure Data Engineer Associate
• AWS Big Data Specialty

Soft Skills

• Strong problem-solving and analytical thinking.
• Excellent communication and collaboration abilities.
• Proactive mindset with ownership of deliverables.
• Passion for learning emerging data technologies.

Why Join Us

• Work with cutting-edge data technologies on large-scale projects.
• Opportunity to shape the organizations data strategy and architecture.
• Collaborative and innovative culture focused on continuous growth.