Senior Data Engineer(AWS Databricks)

Roles and Responsibilities:

• Work with data product managers, analysts, and data scientists to build data/ML pipelines, and data warehouses/lakes for analytics, reporting and prediction.
• Build pipelines and data integration with data feeds/APIs using Apache Airflow & Python.
• Build and own a multi-TB scale data platform on Databricks/Snowflake/AWS.
• Work with data quality and observability tools, and CI/CD frameworks to maintain data engineering excellence.
• Work on data modelling, lineage and data governance.
• Advocate for data architecture across the organisation.

The ideal skills for this role include:

• You have a degree in software engineering, computer science or a similar field and 5+ years of data engineering development experience in distributed data systems/platforms.
• You have strong ETL/ELT knowledge.
• You have a good understanding of data warehouses and big data concepts.
• You are an expert at writing complex SQL queries and scripts.
• You have good experience in using the AWS cloud platform and its ETL services like S3, AWS Glue, Athena, EMR etc.
• You have experience working with the Databricks platform.
• You are comfortable programming in Python and PySpark.
• You are experienced in building data pipelines on the cloud using Airflow (or equivalent).
• Data modelling skills and familiarity with the Kimball methodology.
• You have experience working with Agile/Scrum methodologies.

Additional skills:

• Experience with Big Data technologies, in particular Spark, Hive, and Hadoop.
• Experience in using AWS data services like s3, EMR, Athena, Glue, Redshift, Kinesis, Lambda, EMR, and EKS/ECS.
• Machine learning engineering.