Big Data Engineer

Job Title: Senior Big Data Engineer / Hadoop Spark Lead

Experience: 10-15 Years

Location: Bangalore / Chennai

Role: Senior Technical / Lead Role

Department: IT Data Engineering

Industry: IT Services & Consulting

Job Summary

We are looking for an experienced Senior Big Data Engineer with strong expertise in the Hadoop Ecosystem, PySpark, SQL, Linux and Shell Scripting. The candidate will be responsible for designing, developing and optimizing scalable big data solutions and data processing pipelines.

Key Responsibilities

• Design, develop and maintain large-scale data processing solutions using the Hadoop Ecosystem.
• Develop and optimize data pipelines using PySpark and SQL.
• Work on Linux environments and automate processes using Shell Scripting.
• Handle data processing, transformation, integration and performance optimization.
• Troubleshoot complex data engineering issues and ensure data quality.
• Collaborate with cross-functional teams to deliver scalable data solutions.

Mandatory Skills

• Strong experience in Core Big Data / Hadoop Ecosystem.
• Hands-on experience in PySpark.
• Strong SQL skills.
• Linux / Unix.
• Shell Scripting.

Good-to-Have Skills

• Kubernetes
• DevOps
• Apache Airflow
• MinIO
• Apache Iceberg