Senior/Lead Data Engineer - AWS Glue

Your responsibilities will include developing robust and scalable data pipelines, integrating data sources like SAP HANA and SQL databases, and optimizing pipelines for efficiency. You will also be involved in designing data transformation processes, utilizing SQL and Python for ETL tasks, and ensuring data quality through rigorous testing.

Collaboration and communication will be key aspects of your role as you work closely with data scientists, analysts, and stakeholders to understand data requirements and deliver solutions that meet their needs. You will collaborate with cross-functional teams to implement DataOps practices and improve data lifecycle management.

Monitoring and optimization will be crucial as you monitor data pipeline performance, troubleshoot and resolve data-related issues, and implement monitoring systems to proactively address potential issues. You will also be responsible for maintaining comprehensive documentation of data pipelines, adhering to best practices in data engineering, and staying updated on industry trends.

Required skills for this role include extensive experience with AWS Glue, proficiency in Apache Airflow, strong knowledge of Kafka, advanced SQL skills, proficiency in Python, and experience with SAP HANA. Knowledge of Snowflake, other AWS data services, and big data technologies is preferred. Soft skills such as strong analytical abilities, excellent communication, attention to detail, and the ability to work independently on multiple projects will also be valuable in this role.,