Senior Data Engineer

If working in an environment that encourages you to innovate and excel, not just in professional but personal life, interests you- you would enjoy your career with Quantiphi!

Role: Senior Data Engineer

Experience: 3 to 6 Years

Location: Mumbai/Bangalore

Role & Responsibilities

• Work with cloud engineers and customers to solve for big data problems by developing utilities for migration, storage and processing on AWS Cloud.
• To migrate ETL jobs from the existing AB Initio to AWS Cloud
• Design and build a cloud migration strategy.
• Build tools to ingest and jobs to process several terabytes per day.
• Design and develop next-gen storage and compute solutions for several large customers.
• Communicate with a wide set of teams, including Infrastructure,DevOps, and cloud customers.
• Build advanced tooling for automation, testing, monitoring, administration, and data operations across multiple cloud clusters.
• Use Continuous Integration and Continuous deployment pipeline for

deployment of code to higher environment in client’s environment

• Technical Point of contact between clients and offshore technical team

• 5+ years’ experience of Hands-on in data structures, AWS, spark, SQL and NoSQL Databases
• Experience with AWS - EMR, Redshift, Lambda, DynamoDB, S3, RDS, SQS, Glue
• Experienced in using Airflow for orchestration
• Strong software development skills in Pyspark
• Experience building and deploying cloud-based solutions at scale.
• Experience in developing Big Data solutions (migration, storage, processing)
• Experience in SQL and Query optimisation
• Ability to clearly communicate technical roadmap, challenges and mitigation
• Experience building and supporting large-scale systems in a production environment

• EMR
• Python, Pyspark
• AWS Redshift
• SQL
• S3

• AB Initio ETL
• AWS Glue
• AWS Lambda
• Automation Tools and frameworks
• Familiarity with SDLC concepts and processes