Junior Big Data And Apache Kafka Pipeline Engineer

What role will you play in the team

You will play a foundational role in our core data architecture team by designing, maintaining, and scaling real-time data ingestion pipelines. Your daily contributions will center around streaming massive volumes of enterprise data reliably across distributed clusters using Apache Kafka, Hadoop, and Spark frameworks to empower downstream machine learning and analytical models.

What you will do

You will architect robust stream-processing topologies, configure Kafka brokers, and optimize Spark jobs to process large-scale datasets efficiently. You will collaborate closely with data platform engineers to monitor cluster health, troubleshoot pipeline bottlenecks, execute schema evolutions, and ensure zero data loss across distributed messaging systems.

Key responsibility

• Design, implement, and maintain real-time event-driven data pipelines utilizing Apache Kafka and Spark Streaming technologies.
• Monitor, tune, and scale Hadoop distributed file systems and Spark cluster resources for optimal query and processing performance.
• Integrate diverse structured and unstructured data sources into centralized data lakes using custom ETL workflows and ingestion scripts.
• Troubleshoot latency issues, consumer lag, and broker failures within complex messaging architectures to guarantee high availability.
• Collaborate with data science and machine learning engineering teams to supply clean, synchronized, and feature-ready real-time streams.
• Implement stringent security protocols, access controls, and data encryption standards across all distributed data nodes and pipelines.

Required Qualification and Skills

• Bachelor's degree in Computer Science, Information Technology, Data Engineering, or a related quantitative discipline.
• Hands-on academic or professional experience with Apache Kafka, Apache Spark, Hadoop, and distributed file systems.
• Proficiency in programming languages such as Python, Scala, or Java for pipeline scripting and data manipulation.
• Solid understanding of SQL, relational databases, NoSQL databases, and distributed database management principles.
• Familiarity with containerization tools like Docker and orchestration platforms like Kubernetes for deployment.
• Strong analytical problem-solving abilities, coupled with effective cross-functional communication and teamwork skills.

Looking to get Placed? Try our Placement Guarantee Plan

Get Hired

Benefits Included

• Comprehensive health, medical, and dental insurance coverage for employees and their immediate family dependents.
• Flexible working hours along with a structured hybrid work arrangement to support a healthy work-life integration.
• Annual learning and development stipend dedicated to technical certifications, conferences, and advanced courses.
• Generous paid time off, public holidays, and well-being leaves designed to prevent professional burnout.
• Regular team-building events, hackathons, and structured mentorship programs driven by industry veterans.

A Day in the Life

Your day begins by reviewing real-time monitoring dashboards to check Kafka consumer lag and Spark cluster job performance. You spend the morning optimizing a streaming pipeline script to handle increased data throughput. Post-lunch, you collaborate with the machine learning team to ingest a new data feed, followed by code reviews and documenting pipeline architectures before wrapping up.