Data Engineer - Python

Job Title: Python Data Engineer

Location: Pune / Remote

Experience Level: 38 years

Role Overview

We are looking for a Python-centric Data Engineer who can design and maintain scalable, high-performance data pipelines for large and complex datasets. The ideal candidate brings strong object-oriented Python programming skills, experience with distributed data systems (like Hadoop or Spark) and a mindset for building modular, reusable, and testable data solutions. While exposure to cloud technologies (AWS preferred) is valuable, this role emphasizes deep Python-based data engineering over platform administration.

Roles & Responsibilities

• Design, develop, and optimize data ingestion and transformation pipelines using Python.
• Apply OOP principles to build modular, maintainable, and reusable data components.
• Work with distributed systems (e.g., Hadoop/Spark) for processing large-scale datasets.
• Develop and enforce data quality, testing, and validation frameworks.
• Collaborate with analysts, data scientists, and product teams to ensure data availability and reliability.
• Participate in code reviews, CI/CD workflows, and infrastructure automation to maintain high engineering standards.
• Contribute to ongoing evolution of data architecture and help integrate with cloud-based data ecosystems.

Technical Skills

Core Expertise

Python Programming

• Strong grasp of object-oriented design, modular code practices, and design patterns.
• Experience with libraries/frameworks such as pandas, PySpark, dask, or custom ETL frameworks.
• Proficiency in writing unit-tested, well-documented, and scalable code (pytest, unittest).

Data Engineering & Ecosystem

• Experience in building and maintaining ETL/ELT pipelines using Python.
• Exposure to Hadoop or Spark ecosystems for distributed data processing.
• Solid understanding of SQL and working with relational or analytical databases (e.g., PostgreSQL, Redshift).
• Familiarity with CI/CD pipelines (Jenkins, GitHub Actions) and Infrastructure as Code (Terraform).
• Basic exposure to AWS data services like S3, Lambda, Step Functions, Glue, or DynamoDB.

Qualifications

• Bachelors or masters degree in engineering or technology or related field.
• 36 years of hands-on experience in data engineering or backend development.
• Proven track record of Python-based pipeline development and distributed data processing.
• Strong foundation in data modeling, data quality, and pipeline orchestration concepts.
• Excellent problem-solving and communication skills, with an ownership-driven mindset.

Desired Qualifications

• Experience with Apache Airflow, Luigi, or Prefect for workflow orchestration.
• Exposure to real-time data streaming (Kafka, Kinesis).
• Familiarity with containerization tools (Docker, Kubernetes).
• Knowledge of data governance, cataloguing and metadata management.