Data Lead - Pyspark, Databricks, Apache Spark

Role Summary

This position is well-suited for an experienced data professional with a strong foundation in data engineering and leadership capabilities. As a Data Lead, you will be at the forefront of driving data initiatives, ensuring robust data solutions are effectively designed and executed.

Responsibilities

• Oversee the end-to-end development and optimization of data pipelines utilizing PySpark, Databricks, and Apache Spark technologies.
• Architect and implement effective ETL/ELT processes to facilitate seamless data transformation and integration.
• Collaborate with cross-functional teams, including data scientists and analysts, to gather requirements and deliver scalable data solutions.
• Lead and mentor a team of data engineers, promoting best practices in data management and fostering innovation.
• Implement rigorous testing and validation procedures to maintain high standards of data quality and integrity.
• Stay abreast of advancements in data engineering tools and techniques, recommending enhancements to existing frameworks.

Must-Have Skills

• In-depth expertise in PySpark, Databricks, and Apache Spark.
• Strong command of SQL and thorough knowledge of data modeling methodologies.
• Experience working with Delta Lake alongside conventional data warehousing concepts.
• Familiarity with established data engineering best practices.
• Excellent analytical skills with meticulous attention to detail.
• Proven leadership skills accompanied by effective communication abilities.

Good-to-Have Skills

• Knowledge of cloud platforms relevant to data engineering projects.
• Experience with big data technologies outside of the Apache ecosystem.

Candidates meeting these qualifications are encouraged to apply for this role to lead transformative data projects within our organization.