Junior Big Data And Spark Pipeline Developer

What role will you play in the team

You will play a foundational role in our core Data Science, Analytics & Machine Learning infrastructure team in Bangalore. Your primary objective is to design, construct, and maintain robust data pipelines using distributed computing frameworks. You will bridge the gap between raw unstructured data sources and machine learning model training datasets, ensuring optimal performance, scalability, and high reliability across all data engineering workflows within enterprise operations.

What you will do

You will actively write, optimize, and deploy distributed data processing jobs using Apache Spark, Hadoop, and SQL. Collaborating closely with senior data engineers and machine learning scientists, you will ingest massive volumes of transactional logs and streaming data. Your day-to-day work involves troubleshooting pipeline bottlenecks, refining ETL logic, managing MongoDB and PostgreSQL databases, and validating data integrity to feed downstream predictive models and analytics dashboards seamlessly.

Key responsibility

• Design, develop, and maintain large-scale distributed data pipelines utilizing Apache Spark, Hadoop, and advanced SQL querying techniques to process multi-terabyte datasets daily.
• Integrate diverse data streams from enterprise transactional systems into centralized data lakes and data warehouses using optimized ETL tools and custom Python scripts.
• Monitor, debug, and optimize existing data workflows running on cluster environments to reduce job execution time and minimize resource compute costs.
• Collaborate with machine learning engineers to structure feature stores and deliver clean, transformed training data for production-grade predictive models and neural networks.

• Implement rigorous data quality checks, schema validations, and automated error-handling mechanisms across all ingestion nodes to prevent data corruption.
• Manage database schema changes, indexing strategies, and query performance tuning across PostgreSQL, MySQL, and MongoDB data storage layers.
• Document pipeline architectures, data lineage maps, and operational runbooks to ensure seamless knowledge transfer across the core analytics engineering division.

Required Qualification and Skills

• Bachelor's degree in Computer Science, Statistics, Data Science, Information Technology, or a highly quantitative engineering discipline from a recognized university.
• Demonstrated foundational knowledge of Big Data Technologies, specifically Apache Spark, Hadoop distributed file systems, and distributed computing principles.
• Strong programming proficiency in Python and advanced SQL for complex data manipulation, aggregation, and relational database management systems.
• Hands-on experience working with modern database systems including PostgreSQL, MySQL, and NoSQL databases like MongoDB.
• Familiarity with ETL concepts, data warehousing architecture, version control systems such as Git, and Linux command-line environments.
• Excellent analytical thinking, structured problem-solving abilities, and effective communication skills to collaborate within cross-functional technical teams in Bangalore.

Benefits Included

• Competitive annual compensation package with performance-linked bonuses and annual appraisal reviews based on technical milestone achievements.
• Comprehensive health, medical, and life insurance coverage extending to immediate family members from day one of employment.
• Generous paid time off, casual leaves, maternity and paternity leave policies, and dedicated mental health wellness days.
• Continuous learning allowance for professional certifications, cloud computing credentials, advanced data engineering courses, and technical conference attendance.
• State-of-the-art office workspace located in Bangalore equipped with ergonomic seating, high-performance computing hardware, and collaborative recreation zones.

A Day in the Life

Your day begins by checking cluster health metrics and reviewing automated pipeline execution logs from overnight batch runs in Bangalore. You spend the morning optimizing a slow-running Spark SQL join that processes customer transaction histories. After a quick sync with the data science squad, you write custom Python data transformation scripts, update PostgreSQL index configurations, and push your code through CI/CD pipelines before wrapping up with a technical retrospective.