Data Engineer

We are looking for a Data Engineer to join our data team to solve data-driven critical business problems. You will be responsible for data infrastructure and analytics initiatives. The Data Engineer will collaborate with software developers, database architects, data analysts, data scientists, and platform team on data initiatives and will ensure optimal data delivery architecture is consistent throughout ongoing projects. The right candidate should have hands on in developing a hybrid set of data-pipelines depending on the business requirements.

Responsibilities

• Develop and maintain ETL pipelines to collect, process, and transform data from various sources.
• Implement data modeling techniques to structure and organise data for efficient analysis.
• Align architecture with business requirements and provide solutions that fit best to solve the business problems.
• Identify ways to improve data reliability, efficiency, and quality
• Use data to discover tasks that can be automated.
• Work on AWS cloud infrastructure to store, process, and retrieve data.
• Collaborate with the team to identify opportunities for performance optimisation.

Requirements

• Bachelors degree in Computer Science or equivalent
• 1+ years of relevant work experience as a Data Engineer or in a similar role.
• Advanced SQL knowledge, Data-Modelling, and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases.
• Experience in developing and optimising ETL pipelines, big data pipelines, and data-driven architectures.
• Hands-on experience with AWS services such as S3 EC2 EMR, Glue, etc. AWS Data Analytics certification is preferred.
• Must have strong big-data core knowledge & experience in programming using Spark - Python/Scala
• Hands-on experience in working with real-time data streaming using Kafka
• Experience with orchestrating tools like Airflow or similar
• Build processes supporting data transformation, data structures, metadata, dependency, and workload management.
• A good understanding of Git workflow, Test-case driven development, and using CICD is good to have
• Good to have some understanding of Delta tables

This job was posted by Ravi Mevcha from Oneture Technologies.