Big Data Hadoop And Apache Spark Infrastructure Engineer

What role will you play in the team

You will play a pivotal architectural role within our core data engineering division, focusing entirely on scaling and optimizing massive distributed data processing frameworks. Your core mission involves maintaining fault-tolerant data pipelines using Hadoop and Spark to support intensive machine learning workflows across enterprise data warehouses.

What you will do

You will design, implement, and fine-tune large-scale distributed computing clusters using Apache Hadoop, HDFS, and Apache Spark. You will collaborate closely with data science teams to ingest disparate datasets, optimize memory utilization, and streamline batch and real-time streaming architectures using Apache Kafka and Apache Nifi.

Key responsibility

• Design and maintain robust Apache Hadoop and Apache Spark distributed clusters for high-throughput batch processing.
• Configure and manage Apache Kafka event streaming platforms to ingest and process real-time telemetry data streams.
• Develop and optimize complex ETL data pipelines utilizing Talend and Apache Nifi for seamless data integration.
• Monitor cluster performance, troubleshoot memory bottlenecks, and tune distributed computing jobs for optimal resource utilization.
• Implement enterprise-grade security, data governance, and access controls across HDFS and distributed data lakes.
• Collaborate with data scientists and machine learning engineers to productionize data preprocessing models efficiently.

Required Qualification and Skills

• Bachelor's degree in Computer Science, Data Engineering, or a closely related technical field.
• Minimum of 0 to 1 years of hands-on experience in Big Data Technologies, Hadoop, Spark, and Apache Kafka.
• Strong proficiency in programming languages including Python and SQL for data manipulation and querying.
• Demonstrated practical knowledge of ETL Tools like Talend, Informatica, and Apache Nifi.
• Familiarity with database management systems such as PostgreSQL, MySQL, and MongoDB.
• Understanding of distributed computing concepts, cluster tuning, and cloud-based data lake architectures.

Benefits Included

• Comprehensive health insurance coverage including hospitalization and outpatient care for self and dependents.
• Competitive annual performance-based bonuses tied directly to system optimization milestones.
• Flexible working hours with robust infrastructure support for remote connectivity when needed.
• Sponsored professional certifications in Big Data Technologies, Apache Spark, and Cloud Data Engineering.
• Generous paid time off, maternity/paternity leave, and annual wellness allowances.

A Day in the Life

Your day begins by reviewing cluster health dashboards and resolving overnight Spark execution errors. You will spend the morning optimizing a complex ETL pipeline in Apache Nifi and tuning Kafka consumer lags. In the afternoon, you will meet with machine learning engineers to deploy a large-scale feature engineering script onto the Hadoop cluster, concluding with code reviews.