Senior Data Engineer

Description

• Design, develop, and maintain scalable and reliable data pipelines and platforms on Google Cloud Platform to support enterprise analytics and reporting.
• Collaborate with business and technical teams to understand data requirements and translate them into efficient data engineering solutions.
• Build data ingestion frameworks using Kafka Confluent and integrate data from multiple cloud and on-premises sources.
• Implement ELT and ETL pipelines using PySpark, SQL, Dataform, DBT, and Dataproc for efficient data transformation and processing.
• Design and manage data models, schemas, and data warehousing structures optimized for BigQuery performance and scalability.
• Ensure data quality, consistency, and integrity through effective data validation and monitoring mechanisms.
• Implement data governance practices including data lineage, metadata management, privacy, and compliance.Develop automated monitoring and alerting mechanisms to ensure reliability, availability, and performance of data pipelines.
• Collaborate with cross-functional teams to enable downstream reporting, analytics, and AI/ML model consumption.
• Follow best practices for version control, documentation, and CI/CD workflows using Git and infrastructure-as-code tools.
• Provide operational support for data pipelines and proactively troubleshoot performance issues.
• Continuously optimize data processing, query execution, and resource utilization in GCP environments.
  

Qualifications

• Strong programming skills in
  

Python and
PySpark for large-scale data processing.

• Proficiency in
  

SQL for data manipulation, analysis, and performance tuning.

• Experience with
  

Dataform ,
Dataproc , and
BigQuery for data pipeline development and orchestration.

• Hands-on experience with
  

Kafka and
Confluent for real-time data streaming.

• Knowledge of
  

Cloud Scheduler and
Dataflow for automation and workflow management.

• Familiarity with
  

DBT ,
Machine Learning , and
AI

Looking to get Placed? Try our Placement Guarantee Plan

Get Hired

concepts is a good advantage.

• Understanding of
  

Data Governance principles and implementation practices.

• Experience using
  

Git for version control and
CI/CD pipelines for automated deployments.

• Working knowledge of
  

Infrastructure as Code (IaC) for cloud resource management and automation.
This job posting will remain open a minimum of 72 hours and on an ongoing basis until filled.
Job Information Technology
Primary Location India-Karnataka-Bengaluru
Schedule: Full-time
Travel: No
Req ID: 254883
Job Hire Type Experienced Not Applicable #BMI N/A