Data Engineer

Our client is one of the oldest fintech companies that is taking banking and financial services to all the customers through their online platform. Having served over 50 million customers in the last 15 years, it is enabling over 7mn banking transactions each month, with a network of nearly 2 lac merchants. Using its vast network of merchant outlets, the platform is reaching the lower and mid-income groups who deal in cash, for them to be able to remit money across the country digitally. It now plans to take its unique digital financial solutions to developing markets across the globe. As pioneers of mobile-based payment services in India, they empower Retailers, Individuals and Businesses to have an online presence and earn or save a little extra through the transactions.

As a Data Engineer, you will be responsible for designing, building and integrating data from various sources into data lakes and building large scale data processing pipelines in the context of data analytics projects.

What You Will Do

• Leveraging your deep knowledge to provide technical leadership to take projects from zero to completion
• Architecting, building and maintaining scalable data pipelines and accessing patterns related to permissions and security
• Researching, evaluating and utilising new technologies/tools/frameworks centred around high-volume data processing
• Involving in building and deploying large scale data processing pipelines in a production environment
• Working with data scientists and other engineers to develop data pipelines for model development and productization
• Identifying gaps and implementing solutions for data security, quality and automation of processes
• Providing inputs on right tool options and model designs for use cases
• Identifying gaps and implementing solutions for data security, quality and automation of processes
• Designing scalable implementations of the models developed by our Data Scientists

Desired Candidate Profile

What You Need To Have

• 3+ years strong programming experience in PySpark and Python
• Knowledge in Python, SQL, Spark (Pyspark)
• Exposure to AWS/ Azure cloud tools and services like S3 Athena, Apache Nifi, Apache Airflow
• Analytical and problem-solving skills
• Knowledge in Scrum & code sharing Tech: Git, Jira
• Experience related to processing frameworks such as Spark, Spark Streaming, Hive, Sqoop, Kafka etc
• Deep understanding of measuring and ensuring data quality at scale and the required tooling to monitor and optimise the performance of our data pipelines
• Experience building data pipelines and data-centric applications using distributed storage platforms and shipping data production pipelines sourcing data from a diverse array of sources

Skills:- Python and PySpark