Lead Data Engineer

Job Title : Lead Data Engineer

Location : Bangalore

Responsibilities :

• Continuously monitor data pipeline performance and reliability, optimizing workflows and proactively resolving issues. requirements.
• Continuously monitor data pipeline performance and reliability, optimizing workflows and proactivelyresolving issues.
• Integrate and connect various data sources, including structured and unstructured data from global platforms.
• Identify, evaluate, and integrate new data sources to broaden the scope and improve the quality of available data sets.
• Optimize indexing strategies for handling large-scale, multi-domain datasets efficiently.
• Ensure strict adherence to data quality, consistency, and enterprise compliance standards throughout data processing.
• Develop and implement monitoring and alerting mechanisms to track data pipeline health and SLA compliance.
• Partner with AI/ML engineers, data scientists, and product teams to enable seamless delivery of data-driven solutions.
• Collaborate with cross-functional teams to ensure data engineering initiatives align with strategic project objectives and deadlines.

Qualifications & Skills :

• Bachelor's or Masters degree in Computer Science Engineering or related field, accompanied by 10+years of professional experience in data engineering.
• Proficiency with programming languages including Python, PySpark, and SQL for developing efficient andscalable data solutions.
• Hands-on experience with leading data engineering tools such as Azure DataBricks, Azure DevOps, andVisual Studio Code.
• Strong knowledge of REST APIs, facilitating effective data integration and automation.
• Familiarity with Generative AI concepts and experience integrating Large Language Models (LLM) ishighly desirable.
• Solid understanding of cloud-based data platforms and designing scalable data architectures.