Junior Big Data Hadoop And Spark Pipeline Engineer

What role will you play in the team

As a Junior Big Data Hadoop and Spark Pipeline Engineer, you will serve as a foundational pillar within our core Data Science, Analytics & Machine Learning department in Kolkata. Your primary focus will be designing, developing, and maintaining high-throughput distributed data pipelines. You will collaborate closely with senior data architects and machine learning engineers to ingest, clean, and process massive volumes of structured and unstructured enterprise datasets, ensuring seamless data flow for downstream analytical models and predictive algorithms.

What you will do

You will write and optimize complex Apache Spark jobs, manage Hadoop distributed file systems, and configure Apache Kafka event streams to handle real-time data ingestion. Your daily routines will involve monitoring cluster performance, troubleshooting pipeline failures, writing efficient ETL workflows using Talend and Hive, and implementing robust data governance policies. Furthermore, you will curate structured database stores utilizing PostgreSQL, MySQL, and MongoDB to support rapid querying by data science teams.

Key responsibility

• Develop and maintain scalable batch and real-time data processing pipelines using Apache Spark, Hadoop, and Apache Kafka.
• Design and implement efficient ETL and ELT data integration workflows leveraging tools like Talend and Apache NiFi.
• Optimize large-scale data queries and database schemas across relational and NoSQL platforms including PostgreSQL, MySQL, and MongoDB.
• Monitor distributed computing clusters, diagnose performance bottlenecks, and tune resource allocation for maximum throughput.
• Collaborate with machine learning engineers to format, clean, and deliver enterprise datasets required for feature engineering and model training.
• Implement rigorous data validation, cleansing protocols, and error-handling mechanisms within automated data ingestion pipelines.
• Maintain comprehensive technical documentation of data architecture blueprints, pipeline dependencies, and operational runbooks.

Required Qualification and Skills

• Bachelor's degree in Computer Science, Information Technology, Statistics, or a closely related quantitative discipline.
• 0 to 1 year of hands-on professional or academic experience in big data processing, database management, and pipeline engineering.
• Proficiency in core programming languages including Python and SQL for data manipulation and querying.
• Demonstrated practical knowledge of Big Data Technologies such as Hadoop, Spark, and Apache Kafka.
• Familiarity with database administration and querying concepts across PostgreSQL, MySQL, and MongoDB.
• Understanding of ETL tools like Talend, Informatica, or Apache NiFi for data movement orchestration.
• Strong analytical problem-solving skills, ability to work collaboratively in agile teams, and excellent written communication abilities.

Looking to get Placed? Try our Placement Guarantee Plan

Get Hired

Benefits Included

• Competitive entry-level annual salary package with performance-based appraisal reviews conducted semi-annually.
• Comprehensive health, medical, and life insurance coverage extending to immediate family members.
• Robust professional development stipend allocated for certifications in big data engineering and cloud platforms.
• Flexible working hours with modern ergonomic workstation setups provided for optimal productivity.
• Generous paid time off, casual leave policies, and dedicated mental health wellness days.
• Regular internal technical workshops, hackathons, and mentorship programs led by industry veterans.

A Day in the Life

Your day begins by reviewing real-time monitoring dashboards for your Apache Spark and Hadoop clusters running overnight batch loads. You spend the morning debugging a slow-running ETL pipeline and optimizing SQL queries for a PostgreSQL data warehouse. After a quick stand-up with the data science team, you dedicate the afternoon to writing Python scripts for automated data validation and testing a new Apache Kafka streaming topic before its deployment to staging.