Data Engineer

Company:IT Services Organization

Key Skills: Azure Data Factory, Azure Databricks, PySpark, SQL.

Responsibilities:

• Design, develop, and maintain data pipelines: Utilize your expertise in Azure Data Factory (ADF), PySpark, SQL, and Databricks to create efficient and reliable data pipelines to meet business needs.
• Collaborate with stakeholders: Work closely with data analysts, data scientists, and other team members to understand business requirements and translate them into effective technical solutions.
• Data quality and validation: Implement data quality checks and validation processes to ensure the accuracy and integrity of the data flowing through the pipelines.
• Data security and compliance: Implement appropriate security measures and ensure compliance with data governance policies and regulations.
• Performance tuning and optimization: Continuously monitor and fine-tune data pipelines to optimize their performance and efficiency.
• Documentation: Create and maintain comprehensive documentation of data pipelines and processes to ensure smooth knowledge transfer and future maintenance.
• User support and issue resolution: Provide support to users and troubleshoot any issues related to data pipelines to ensure seamless data integration and availability.
• Stay up-to-date with the latest technologies: Keep abreast of the latest developments in Azure data engineering technologies and trends and apply relevant advancements to enhance our data engineering processes.

Required Experience:

• 3-8 years of hands-on experience in Azure data engineering, with a track record of successfully delivering data solutions.
• Technical Skills: Strong understanding of Azure data engineering concepts and proficiency in developing data pipelines using Azure Data Factory (ADF), PySpark, SQL, and Databricks.
• Data modelling and ETL: Experience with data modelling, data warehousing, and ETL processes.
• Agile methodologies: Familiarity with Agile software development methodologies and the ability to adapt to changing project requirements.
• Problem-solving and analytical skills: Excellent problem-solving abilities and a keen analytical mindset to address complex data engineering challenges.
• Communication and collaboration: Effective communication and interpersonal skills to effectively work with cross-functional teams and stakeholders.
• Independence and teamwork: Ability to work both independently and collaboratively in a team-oriented environment.

Qualifications:

• Any Graduation degree in information technology, computer science & technology.