Senior Data Engineer (Databricks AWS)

Roles and Responsibilities:

• Work with data product managers, analysts, and data scientists to build data/ML pipelines, and data warehouses/lakes for analytics, reporting, and prediction.
• Build pipelines and data integration with data feeds/APIs using Apache Airflow & Python.
• Build and own a multi-TB scale data platform on Databricks/Snowflake/AWS.
• Work with data quality and observability tools, and CI/CD frameworks to maintain data engineering excellence.
• Work on data modeling, lineage, and data governance
• Advocate for data architecture across the organization

The ideal skills for this role include:

• You have a degree in software engineering, computer science, or a similar field and 5+ years of data engineering development experience in distributed data systems/platforms.
• You have strong ETL/ELT knowledge
• You have a good understanding of data warehouses and big data concepts
• You are an expert at writing complex SQL queries and scripts.
• You have good experience in using the AWS cloud platform and its ETL services like S3, AWS Glue, Athena, EMR, etc
• You have experience working with the Databricks platform.
• You are comfortable programming in Python and PySpark.
• You are experienced in building data pipelines on the cloud using Airflow (or equivalent).
• Data modeling skills and familiarity with the Kimball methodology
• You have experience working with Agile/Scrum methodologies.

Additional skills:

• Experience of Big Data technologies, in particular Spark, Hive, and Hadoop.
• Experience in using AWS data services like s3, EMR, Athena, Glue, Redshift, Kinesis, Lambda, EMR, EKS/ECS
• Machine learning engineering