Data Engineer - Pyspark, Azure, Python

• Role Summary
• In the first few months, you will establish and optimize data pipelines that enhance our analytics capabilities and deliver actionable insights.
• Responsibilities

• Design and develop efficient data ingestion workflows to process large datasets from diverse sources.
• Collaborate with data scientists and analysts to define data requirements, ensuring that solutions meet business needs.
• Implement data quality checks and validation processes to maintain the integrity of data throughout its lifecycle.
• Leverage Azure cloud services for deploying and managing scalable data solutions.
• Create comprehensive documentation for all data engineering processes and systems to facilitate knowledge sharing.
• Stay informed about advancements in data engineering technologies and methodologies, particularly in the Azure cloud ecosystem.

Must-Have Skills

• Proficient in PySpark for efficient data transformation and processing tasks.
• Extensive experience with Azure cloud services, especially in relation to data storage solutions.
• Strong programming expertise in Python, emphasizing data manipulation and analytical techniques.
• Familiarity with Databricks for executing collaborative data projects and enhancing analytics capabilities.
• Exceptional problem-solving skills, enabling effective troubleshooting and optimization of data workflows.

Good-to-Have Skills

• Experience with additional data processing frameworks or tools beyond PySpark.
• Knowledge of machine learning concepts that could enhance the utility of the developed pipelines.

This role offers a chance to deepen your expertise in cloud-based data solutions while contributing to innovative projects that drive business intelligence initiatives forward.