Data Engineer (AWS/ Azure)

As an AWS Data Engineer, you will be responsible for leading development activities for the Data engineering team. You will collaborate with other teams such as application management and product delivery, working closely with technical leads, product managers, and support teams. Your role will involve providing guidance to the development, support, and product delivery teams. Additionally, you will lead the implementation of tools and technologies to drive cost-efficient architecture and infrastructure.

On the other hand, as an Azure Data Engineer, your responsibilities will include creating and maintaining optimal data pipelines, assembling large, complex data sets that meet business requirements, and identifying opportunities for process improvements and automation. You will develop data tools for analytics and data science teams to optimize product performance and build analytics tools for actionable insights into business metrics. Collaboration with stakeholders from various teams will also be essential to address data-related technical issues and support data infrastructure needs.

The ideal candidate for the AWS Data Engineer position should have experience with AWS services like S3, Glue, SNS, SQS, Lambda, Redshift, and RDS. Proficiency in programming, especially in Python, is required, along with strong skills in designing complex SQL queries and optimizing data retrieval. Knowledge of spark, Pyspark, Hadoop, Hive, and spark-Sql is also essential.

For the Azure Data Engineer role, candidates should have experience with Azure cloud services and developing Big Data applications using Spark, Hive, Sqoop, Kafka, and Map Reduce. Familiarity with stream-processing systems such as Spark-Streaming and Strom will be advantageous.,