Senior Data Engineer

About the role

We are seeking Senior Data Engineers to join our data engineering team, who are passionate about data & analytics. The ideal fit for the role will have a strong background in handling large volumes of data with Apache Spark to build , enhance bespoke systems to harness data .The key focus of the position is to build or maintain systems to capture and store data on behalf of the business.

Key Responsibilities

• Design, develop, and maintain ETL processes and data pipelines using AWS Glue with Pyspark.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver high-quality data solutions.
• Optimize and tune data pipelines for performance and scalability.
• Ensure data quality and integrity through robust testing and validation processes.
• Implement data governance and security best practices.
• Monitor and troubleshoot data pipelines to ensure continuous data flow and address any issues promptly.
• Stay up-to-date with the latest trends and technologies in data engineering.

Required Qualification

• B.E/B.Tech, preferably in Computer Science Engineering with relevant work experience.
• 7+ years of experience in handling data and designing ETL pipelines with mandatory 4+ years experience in writing Spark.
• Experience with AWS services such as Glue, Athena, S3, and Redshift is a plus.
• Good to have exposure to data modeling, data analytics, and design in both batch processing and real-time streaming.
• Solid understanding of data mapping, data processing patterns, distributed computing, and building applications for real-time and batch analytics
• Strong programming skills in design and implementation using Python, PySpark, SQL
• Good exposure in database architecture with SQL.
• Experience with multiple file formats like Avro, Parquet, ORC, and JSON.
• Developing, constructing, testing, and maintaining architectures for data lakes, data pipelines, data warehouses, and large-scale data processing systems on Databricks.