Junior Big Data And Apache Kafka Stream Processing Engineer

What role will you play in the team

You will play a critical foundational role within our enterprise data infrastructure engineering team, focusing on ingestion pipelines and real-time event streaming systems. Your primary objective will be to design, develop, and maintain robust data streaming architectures utilizing Apache Kafka and distributed computing frameworks. You will bridge the gap between raw data generators and downstream machine learning models by ensuring reliable, high-throughput message delivery and stream processing across massive enterprise datasets.

What you will do

You will actively build and optimize distributed data ingestion pipelines, configure Kafka brokers, topics, and consumer groups, and integrate big data ecosystems with relational and non-relational database management systems. Your day-to-day engineering duties will involve writing high-performance transformation scripts, monitoring stream latency, debugging distributed pipeline bottlenecks, and collaborating closely with data scientists to deliver clean, production-ready streaming features for predictive analytical models.

Key responsibility

• Design, implement, and maintain real-time data pipelines utilizing Apache Kafka, ZooKeeper, and Spark Streaming components.
• Develop scalable ETL and data ingestion jobs to process terabytes of incoming structured and unstructured telemetry data.
• Configure and manage Kafka producers, consumers, and cluster parameters to ensure fault-tolerance and high availability.
• Integrate big data storage layers with PostgreSQL, MySQL, and Hadoop distributed file systems for long-term retention.
• Monitor pipeline performance, identify processing bottlenecks, and tune JVM garbage collection and network buffers.
• Write comprehensive unit and integration tests for all streaming components using automated CI/CD frameworks.
• Collaborate with cross-functional teams of data architects and machine learning engineers to align stream schemas.

Required Qualification and Skills

Candidates must possess a Bachelor's degree in Computer Science, Information Technology, or a related quantitative discipline with hands-on technical competencies. Essential technical requirements include strong proficiency in Python and SQL for data manipulation. Hands-on experience with Apache Kafka, Hadoop, and Spark Big Data Technologies is mandatory. Working knowledge of relational databases such as PostgreSQL and MySQL, alongside experience with ETL Tools like Talend or Informatica, is strictly required. Strong debugging skills, understanding of distributed systems architecture, and familiarity with Linux environment management are vital.

Benefits Included

• Comprehensive health, medical, and life insurance coverage for employees and immediate family members.
• Competitive performance-based annual bonuses linked directly to engineering milestone achievements.
• Sponsored professional certifications in Big Data Engineering, Apache Kafka, and Cloud Infrastructure.
• Flexible work arrangements with standard work from office protocols and ergonomic workspace allowances.
• Generous paid time off, maternal and paternal leave policies, and annual wellness stipend packages.

A Day in the Life

Your typical workday begins by reviewing stream processing dashboards and investigating overnight Kafka cluster lag alerts. You will spend the morning writing Python scripts to parse incoming JSON payloads into structured PostgreSQL tables. In the afternoon, you will participate in an agile sprint planning session, refactor a Spark streaming transformation job with senior architects, and test a newly deployed data ingestion pipeline in a staging environment.