Senior Data Engineer

Senior Data Engineer

Experience:

10+ Years

Location:

Gurugram (Hybrid 2 Days Work from Office)

Preference:

Hybrid profiles are strongly preferred

Why Join Us?

• Opportunity to work on Capital Markets and Investment Banking projects.
• Exposure to enterprise-scale platforms used by leading global financial institutions.
• Work with cutting-edge Azure Cloud, Databricks, Spark, Delta Lake, and modern data engineering technologies.
• Hybrid work model with a collaborative and learning-focused culture.
• Opportunity to lead large-scale data platform initiatives and architectural decisions.
• Continuous learning and professional growth in a fast-paced environment.

Preferred Domain Experience / Areas of Opportunity

• Experience in Capital Markets, Investment Banking, Asset Management, or Financial Services is preferred.

Role Overview

As a Senior Data Engineer , you will:

• Work directly with global financial services clients.
• Lead the design, development, and operation of enterprise-scale data platforms on Azure Databricks.
• Build and manage secure, scalable, and auditable data pipelines from ingestion through analytics-ready datasets.
• Drive best practices around data governance, lineage, quality controls, and operational excellence.
• Collaborate with Product, Engineering, AI, and Infrastructure teams to deliver high-quality data solutions

Key Responsibilities

• Lead end-to-end pipelines: secure file ingest (e.g. SFTP) through landing, history (Delta), and curated layers.
• Own Databricks delivery: notebooks, Workflows/Jobs, asset bundles, parameters, concurrency, and retries.
• Implement medallion-style layers (landing -> staging -> curated->analytics) with partitioning, checksums, and metadata sidecars.
• Maintain end-to-end auditability (correlation IDs, trigger and completion records) across the pipeline.
• Partner with SRE on lake/SFTP storage, identity, and secrets for Databricks and related services.
• Enforce data quality, schema evolution, lineage, cost control, and runbooks for pipeline incidents.
• Provide curated, quality-assured datasets from the lake and SQL warehouse to product and AI teams for use in RAG and embedding workflows, under agreed data contracts covering schema, metadata, access classifications and refresh frequency; ownership of RAG corpora and vector stores sits with those teams.

Must have Experience

• 10+ years data engineering; led production platforms end to end.
• Expert Azure Databricks (Spark, Delta, Workflows/Jobs, asset bundles).
• Hands-on data lake design and SFTP or file-landing ingest patterns.
• Strong SQL + Python; orchestration and CI for data code.
• Governance: isolation, RBAC, auditability, PII handling.
• Investment data and data modelling.

Nice to Have

• Familiarity with RAG / vector-search consumers (not required to build).
• dbt / Delta Live Tables; Event Hubs / CDC.
• Databricks SQL or BI serving patterns.