Data Engineer

– Senior Data Engineer

Duration - 12months + extendable

Role Overview

We are looking for a Senior Data Engineer with 5-8 years of strong experience in building scalable, cloud native data platforms and high performance data pipelines. The role involves designing, developing, and optimizing batch and real time data processing systems, supporting analytics, AI/ML, and enterprise use cases across regulated and non regulated domains (Banking, Fintech, Consulting, SaaS)

Key Responsibilities

Data Engineering & Platform Development

• Design, develop, and maintain end to end data pipelines for batch and real time processing.
• Build scalable ETL/ELT frameworks using Python and SQL.
• Implement and manage workflow orchestration using tools such as Airflow.
• Design efficient data models, schemas, and transformation layers to support analytics and downstream consumers.
• Develop ingestion pipelines from diverse sources such as APIs, databases, event streams, and external systems.

Streaming & Performance Optimization

• Build and optimize real time streaming pipelines using messaging platforms (e.g., Kafka).
• Tune pipelines for high throughput, low latency, cost efficiency, and reliability.
  
• Implement monitoring, alerting, and logging for data platforms to meet SLAs.

Cloud, DevOps & Engineering Best Practices

• Work with cloud platforms mainly on Azure for data storage, processing, and orchestration. Good to have experience on AWS or GCP.
• Use containerization (Docker) and CI/CD pipelines for deployment and version control.
• Follow software engineering best practices including code reviews, testing, and documentation.
• Ensure data quality, validation, and governance across pipelines.

AI / Advanced Data Use Cases

• Support AI/ML ready datasets, Feature Stores, and MLOps workflows.
• Build or integrate LLM based or AI enabled pipelines (e.g., document processing, ETL automation, RAG based solutions).
• Collaborate closely with Data Scientists, ML Engineers, and Analytics teams.

Collaboration & Ownership

• Translate business requirements into scalable technical solutions.
• Work with cross functional teams including Product, Analytics, and Business stakeholders.
• (For senior/lead roles) Mentor junior engineers and contribute to architectural decisions.

Must Have

• 5–8+ years of experience in Data Engineering
• Strong programming experience in Python
• Solid expertise in SQL and data transformations
• Hands on experience with ETL/ELT pipelines
• Experience with cloud platforms (AWS / Azure / GCP)
• Exposure to real time data streaming (Kafka or equivalent)
• Experience with Airflow or similar orchestration tools
• Strong understanding of data modeling and data warehousing principles

Good to Have

• Experience with Snowflake, BigQuery, Redshift, Delta Lake, dbt, Spark/PySpark
• Exposure to MLOps, Feature Stores, MLflow
• Experience with LLMs, Generative AI, or RAG architectures
• Experience in Banking, Fintech, Consulting, or Enterprise SaaS

Prior experience in technical leadership or client facing roles