Data Engineer (Python GCP)

Title: Data Engineer (Python+ GCP)

Experience: 6+ years

Location: Hyderabad

Notice period: Immediate -upto 30 days

Description:

We are seeking a talented and experienced Python Developer with expertise in Google Cloud Platform (GCP), Big Data technologies, and Data Lake architecture. As a member of our team, you will be responsible for designing and developing an application that leverages Apache Airflow to connect to Google BigTable and perform data validation tasks. This is an exciting opportunity to work on cutting-edge technology and contribute to the success of our data processing initiatives.

Responsibilities:

• Collaborate with cross-functional teams to understand data validation requirements and translate them into technical specifications.
• Design and develop scalable and efficient data processing workflows using Apache Airflow. Integrate the application with Google BigTable, ensuring optimal performance and data integrity. Implement data validation strategies and algorithms to ensure the accuracy and quality of processed data.
• Troubleshoot and resolve issues related to data processing, connectivity, and validation.
• Optimize and fine-tune the application for performance and reliability.
• Stay up-to-date with the latest developments in GCP, Big Data, and Data Lake technologies, and propose innovative solutions to enhance the application.

Qualification:

• Bachelors degree in Computer Science, Engineering, or a related field. Masters degree is a plus. Proven experience as a Python Developer, with a strong understanding of Python programming concepts and best practices.
• Hands-on experience with Google Cloud Platform (GCP) services, including BigTable, Dataflow, and Pub/Sub.
• Familiarity with Data Lake architecture and related technologies (e.g., Google Cloud Storage, Hadoop, Spark).
• Previous experience working with Apache Airflow for designing and orchestrating data workflows. Solid understanding of data validation methodologies and techniques.
• Proficiency in SQL and database concepts.
• Strong problem-solving skills and the ability to troubleshoot complex technical issues.
• Excellent communication and teamwork skills.
• Attention to detail and a commitment to producing high-quality code and deliverables.
• Experience with version control systems (e.g., Git) and agile development methodologies.