Work from Office
Job Designation / Role: Data Engineer (Python - Spark - SQL)
Location: Ahmedabad, Hyderabad
We are looking for a highly motivated and experienced Data Engineer to join our team of data experts. The ideal candidate will have a strong background in designing, developing, and maintaining data pipelines and ETL processes using technologies such as Snowflake, Databricks, Apache Spark, and Hadoop. As a Data Engineer, you will work closely with the Lead Data Engineer and Data Architect to implement end-to-end data solutions, build and maintain data pipelines, and ensure the quality and integrity of our organizations data.
Roles & Responsibilities:
- Collaborate with the Lead Data Engineer and Data Architect to design and implement end-to-end data solutions
- Develop and maintain data pipelines and ETL processes using Python, Spark, and SQL
- Ensure data quality and integrity by developing and implementing data validation and testing strategies
- Implement data security measures to protect the confidentiality, integrity, and availability of our organizations data
- Continuously monitor and evaluate the performance of data systems and make recommendations for improvements
- Stay up-to-date with the latest data technologies and trends and provide recommendations for their adoption
- Work collaboratively with other teams to integrate data from different sources
- Participate in code and design reviews to ensure the quality of data pipelines and ETL processes
- Develop and maintain documentation for data pipelines, ETL processes, and data models
- Troubleshoot and resolve data-related issues in a timely manner
Skills & Experience Required:
- 4+ years of experience in data engineering and data architecture
- 3+ years of experience as a Python developer.
- Work experience as a Python Developer and Expertise in at least one popular Python framework (like Django, Flask, or Pyramid)
- Writing scalable code using Python programming language.
- A deep understanding and multi-process architecture and the threading limitations of Python
- Good to have Spark knowledge, PySpark, Azure Data Bricks understanding.
- Work experience with pandas
- API debugging skills
- Strong expertise in Python and SQL
- Strong knowledge of data technologies, such as Snowflake, Databricks, Apache Spark, Hadoop, Dbt, Fivetran, Azure Data Factory
- Experience designing and developing end-to-end data solutions, data pipelines, and ETL processes
- Experience with data modelling is a plus
- Excellent problem-solving and analytical skills
- Ability to work independently and as part of a team
- Experience working in an Agile environment
- Knowledge and experience in developing software using agile methodologies.
- Experience working with a globally distributed team.
- Skilled in building relationships with clients and in practice development activities.
- Excellent written and oral communication skills; Ability to communicate effectively with technical and non-technical staff.