Data Engineer

Job Title: Senior Data Engineer (AWS Glue / PySpark)

Experience: 8+ Years

Location: Hyderabad / Chennai

Mode of Work: Hybrid (Minimum 2 days from office)

:

We are looking for a highly skilled Senior Data Engineer with strong experience in designing and implementing scalable data pipelines using AWS Glue, PySpark, and related AWS services. The ideal candidate will have deep expertise in building real-time and batch ETL solutions, optimizing Spark jobs, and managing data infrastructure on AWS.

Key Responsibilities:

• Design, develop, and maintain ETL pipelines using AWS Glue, PySpark, and Lambda functions.
  
• Troubleshoot and resolve Out of Memory (OOM) and performance issues in Spark jobs.
  
• Develop and optimize real-time ETL processes using AWS Lambda.
  
• Implement and manage infrastructure using Terraform and CloudFormation templates.
  
• Work with various AWS services such as RDS, Redshift, IAM, and CloudWatch for data processing and monitoring.
  
• Write efficient SQL queries and manage complex data transformations.
  
• Collaborate with cross-functional teams to design data models, ensure data quality, and support analytical and reporting needs.
  
• Apply strong understanding of database and data warehouse concepts to design robust, scalable data architectures.
  

Required Skills:

• Hands-on experience with AWS Glue and Apache Spark architecture.
  
• Strong proficiency in PySpark and Python programming.
  
• Proven experience handling Spark OOM issues and performance tuning.
  
• Expertise in AWS Lambda, Terraform, CloudFormation, RDS, Redshift, IAM, and CloudWatch.
  
• Strong SQL and data modeling skills.
  
• Solid understanding of data warehouse and relational database concepts.
  
• Ability to work in a hybrid environment and collaborate effectively with global teams.