Job Designation / Role: Data Engineer (Python - Spark - SQL)

Location: Ahmedabad, Hyderabad

Role Objective:

We are looking for a highly motivated and experienced Data Engineer to join our team of data experts. The ideal candidate will have a strong background in designing, developing, and maintaining data pipelines and ETL processes using technologies such as Snowflake, Databricks, Apache Spark, and Hadoop. As a Data Engineer, you will work closely with the Lead Data Engineer and Data Architect to implement end-to-end data solutions, build and maintain data pipelines, and ensure the quality and integrity of our organizations data.

Roles & Responsibilities:

• Collaborate with the Lead Data Engineer and Data Architect to design and implement end-to-end data solutions
• Develop and maintain data pipelines and ETL processes using Python, Spark, and SQL
• Ensure data quality and integrity by developing and implementing data validation and testing strategies
• Implement data security measures to protect the confidentiality, integrity, and availability of our organizations data
• Continuously monitor and evaluate the performance of data systems and make recommendations for improvements
• Stay up-to-date with the latest data technologies and trends and provide recommendations for their adoption
• Work collaboratively with other teams to integrate data from different sources
• Participate in code and design reviews to ensure the quality of data pipelines and ETL processes
• Develop and maintain documentation for data pipelines, ETL processes, and data models
• Troubleshoot and resolve data-related issues in a timely manner

Skills & Experience Required:

• 4+ years of experience in data engineering and data architecture
• 3+ years of experience as a Python developer.
• Work experience as a Python Developer and Expertise in at least one popular Python framework (like Django, Flask, or Pyramid)
• Writing scalable code using Python programming language.
• A deep understanding and multi-process architecture and the threading limitations of Python
• Good to have Spark knowledge, PySpark, Azure Data Bricks understanding.
• Work experience with pandas
• API debugging skills
• Strong expertise in Python and SQL
• Strong knowledge of data technologies, such as Snowflake, Databricks, Apache Spark, Hadoop, Dbt, Fivetran, Azure Data Factory
• Experience designing and developing end-to-end data solutions, data pipelines, and ETL processes
• Experience with data modelling is a plus
• Excellent problem-solving and analytical skills
• Ability to work independently and as part of a team
• Experience working in an Agile environment
• Knowledge and experience in developing software using agile methodologies.
• Experience working with a globally distributed team.
• Skilled in building relationships with clients and in practice development activities.
• Excellent written and oral communication skills; Ability to communicate effectively with technical and non-technical staff.