Lead Data Engineer With Advanced Python & SQL Skills For A Global B2B AI/ML SaaS - Up To 55L ESOPs

Hiring for: A Global B2B SaaS company, using AI/ML to optimise demand and supply processes for retail, manufacturing, FMCG etc.

Experience: 7 to 10 years

Salary: Up to 55L + ESOPs

Important: This role is 80%Individual Contributor and 20% Team Building / Training / Mentoring

Role & Responsibilities:

• Architect the DE pipelines with scalability, configurability, and security
• Develop and deploy scalable and configurable data engineering pipelines
• Lead the data function and set out data strategy and roadmap
• Manage stakeholder relationship at all levels and understand and align their data requirements
• Lead the development, publication and maintenance of design documents and roadmaps
• Review, audit and improve the Data Engineering code base and pipelines
• Support maintenance of deployed pipelines (bug-fixes and performance improvement)
• Participate in organization building activities (team mentoring, cross team training & hiring

Must Have:

• Data structure and algorithm
• Data modelling and design
• Excellent Coding skills in Python and SQL
• Excellent understanding of Pandas API
• Experience of writing/maintaining Spark code base in a production environment
• Experience in orchestration technologies like Airflow, Kubeflow etc.
• Experience on running ETL pipelines from a wide variety of sources, both batch & streaming, using latest data frameworks and technologies with real time monitoring and alerting
• Essential experience of working with distributed systems software development
• Critical experience in performance optimization for both data loading and ingestion
• Know-how of critical developer’s toolkit such as Linux, GitHub, Dockers, VSCode, Jupyter
• Ability to work in a fast-paced and deadline driven environment
• Experience of working on public clouds (AWS / Azure)