Senior Data Engineer

★ Hiring for Disney, HQ ★

Job Summary:

We are seeking a Senior Data Engineer for Disney with extensive experience in processing data pipelines and handling common issues that arise in such environments. The ideal candidate will possess in-depth knowledge of data pipeline technologies, including Airflow, Databricks, and Snowflake. They will be responsible for supporting and optimizing large-scale distributed pipelines while quickly triaging and resolving any issues that may arise. The candidate should be well-versed in navigating logs, analyzing metrics, and following runbooks to ensure efficient pipeline operations. Additionally, the candidate will be expected to write PySpark or Spark Scala code during version upgrades, ensuring backward compatibility and smooth transition. The role also involves conducting proof-of-concept (PoC) projects to explore innovative solutions. While expert-level coding skills are not mandatory, the candidate should have a strong understanding of coding principles and a willingness to learn and grow.

Duties and Responsibilities:

• Utilize extensive experience in processing data pipelines to support large-scale distributed systems based on Airflow, Databricks, and Snowflake.
• Apply in-depth knowledge of data pipeline technologies to quickly triage and resolve issues within the ecosystem.
• Navigate logs, interpret metrics, and follow runbooks to identify and address any pipeline-related concerns.
• Ensure the seamless operation of existing pipelines during version upgrades, identifying and resolving backward compatibility issues.
• Write efficient PySpark or Spark Scala code to enhance pipeline functionalities or accommodate version upgrades.
• Conduct PoC projects to evaluate new technologies, optimize processes, and improve pipeline performance.
• Demonstrate strong troubleshooting skills and promptly resolve issues to minimize disruption.
• Collaborate closely with cross-functional teams to gather requirements and provide technical expertise.
• Stay updated with the latest industry trends and best practices in data pipeline engineering.

Skills and Qualifications:

• Extensive experience in processing data pipelines and handling common issues that arise.
• Strong knowledge of Airflow, Databricks, Snowflake, and Spark technologies.
• Proven ability to quickly triage and resolve issues within the data pipeline ecosystem.
• Proficiency in navigating logs, analyzing metrics, and following runbooks for efficient troubleshooting.
• Experience ensuring smooth operation during version upgrades, addressing backward compatibility concerns.
• Proficiency in writing PySpark or Spark Scala code for pipeline enhancements and version upgrades.
• Solid problem-solving and analytical skills.
• Strong understanding of coding principles with a willingness to learn new languages or frameworks.
• Excellent communication and collaboration skills.
• Bachelors degree in Computer Science, Engineering, or a related field (or equivalent work experience).