Data Engineer

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks, PySpark, and Apache Spark.
• Administer and manage Databricks workspaces, clusters, job configurations, and cluster policies.
• Build efficient ETL workflows to ingest, transform, and load data from multiple structured and unstructured data sources.
• Optimize Spark jobs for performance, reliability, and cost efficiency.
• Implement data quality checks, monitoring, and alerting across pipelines.
• Work with stakeholders to understand data requirements and translate them into technical solutions.
• Support implementation of data governance, ABAC/RBAC policies, and security best practices.
• Develop reusable data frameworks, automation scripts, and libraries to improve engineering productivity.
• Collaborate with analytics, ML, BI, and platform teams to ensure seamless integration and data availability.
• Troubleshoot production issues related to pipelines, clusters, and Spark jobs.
  

Required Skills & Experience

• 5+ years of experience as a Data Engineer or similar role.
• Strong hands-on experience with Databricks, PySpark, and Apache Spark.
• Proficiency in Python, SQL, and distributed data processing.
• Experience with data lakes, Delta Lake, and cloud platforms (Azure/AWS/GCP).
• Familiarity with CI/CD, version control (Git), and DevOps pipelines.
• Strong understanding of data modelling, warehousing concepts, and performance tuning.
• Experience working with REST APIs and ingestion frameworks is a plus.