Lead Data Engineer - Big Data Technologies

Description

Key Responsibilities :

• Extract and process data from multiple databases, APIs, and cloud platforms by establishing strong source connections.
• Design, develop, and maintain ETL pipelines for efficient data transformation and ingestion.
• Write and optimize complex SQL queries for data extraction, validation, and reporting.
• Utilize PySpark and BigData technologies (Hadoop, Spark, Databricks, etc.) to process and analyze large datasets.
• Perform data analysis using Python (Pandas, NumPy, etc.) to uncover trends, patterns, and insights.
• Develop and maintain interactive dashboards and reports in Power BI (good to have).
• Work with cross-functional teams to understand business requirements and translate them into data-driven solutions.
• Experience in Data warehouse schema creation and optimization
• Ensure data accuracy, security, and compliance with company policies.
• Present analytical findings and recommendations to stakeholders in a clear and actionable manner.

• Minimum 5+ Years of experience in data analytics, data engineering, or business intelligence.
• Strong expertise in SQL (writing, optimizing, and managing complex queries).
• Hands-on experience in ETL processes and building/maintaining data pipelines.
• Experience working with Big Data technologies (Hadoop, Spark, PySpark, Databricks, etc.).
• Proficiency in Python for data analysis (Pandas, NumPy, etc.).
• Experience of working on Fabric
• Experience of working on Databrick
• Experience in connecting to and processing data from multiple data sources (databases, APIs, cloud platforms).
• Experience of working on Cloud platforms ( Azure /AWS / GCP)
• Experience of delivering the Projects
• Experience of requirement understanding
• Experience of preparing estimations and creating POCs
• Experience of working with different Project domains.
• Proficiency in delivering High quality codes to clients
• Proficiency in Data warehouse schema creation.
• Experience of client communication.
• Strong analytical and problem-solving skills.
• Experience of mentoring and managing the team.

(ref:hirist.tech)