Data Engineer

Job role: Data Engineer

Location: Hyderabad/Chennai (Hybrid)

Employment Type: Full-time

Team: Data Engineering

No-of Positions: 4

JD:

• Hands-on expertise in GCP data stack: BigQuery, Dataflow (Apache Beam), Dataproc, Cloud Storage, Pub/Sub, Cloud Composer (Airflow).
• Strong Spark (PySpark or Scala) for batch processing on Dataproc.
• Solid Airflow DAG design (idempotent tasks, backfills, retries, SLAs).
• Advanced SQL and data modeling (star/snowflake, slowly changing dimensions, partition strategies).
• Proficiency in Python (preferred) or Scala/Java for data engineering.
• Experience with Git and CI/CD (Cloud Build/GitHub Actions/GitLab CI).
• Familiarity with security & governance on GCP (IAM, service accounts, secrets management, VPC-SC basics).
• Strong debugging skills and ownership mindset; clear communication with technical and non-technical stakeholders.
  Key Responsibilities
• Design, develop, and maintain ELT/ETL pipelines on GCP using Dataflow/Beam, Dataproc/Spark, and Airflow/Composer.
• Model and optimize datasets in BigQuery (partitioning, clustering, materialized views, UDFs).
• Build streaming and near-real-time ingestion using Pub/Sub, Dataflow, and CDC where applicable.
• Implement data quality checks, validation frameworks, and SLAs; monitor pipelines via Cloud Monitoring/Logging.
• Optimize performance and cost across GCS, Dataproc autoscaling, and BigQuery slot usage.
• Contribute to standards for coding, CI/CD, observability, and documentation; perform code reviews.
• Partner with Analytics/BI/ML teams to productize datasets and ensure strong data contracts.
• Support production operations, including on-call rotations for critical pipelines (as needed).
  Good-to-Have
  
• Snowflake (migrations to/from BigQuery, performance tuning, tasks/streams).
• dbt for transformations and testing; Great Expectations or similar DQ frameworks.
• Terraform for infrastructure as code on GCP.
• Kafka or additional streaming tooling; Cloud Run/Functions for glue services.
• BI exposure (Looker/Looker Studio, Tableau, or Power BI).
• GCP Professional Data Engineer certification.