Senior Data Engineer

Department Icon Data Science Analytics & Machine Learning
Posted: 1 year ago
0-1 years
Jaipur, Rajasthan
Data Science Analytics & Machine Learning
work from office
Full Time

Posted: 1 year ago
Let Jobaaj One apply for you

An AI agent that applies to roles like this on employers' own sites. 25 applications free, no card needed.

Job Description
Similar Jobs
Please verify your account first! Send OTP

Posted over 15 days ago, so the role may have closed. Confirm on the employer's page when you apply, or see similar jobs.

Job Description

Role Overview

Design, build, and optimize large-scale, production-grade data pipelines and analytics platforms on Azure, leveraging Databricks, Synapse, and the broader Microsoft data ecosystem.

Deliver business-critical data assets for analytics, BI, and AI/ML initiatives.

Key Technical Responsibilities

  • Architect modern data lakes using Azure Data Lake Storage Gen2 for batch and streaming workloads
  • Build and maintain scalable ETL/ELT pipelines using Azure Data Factory and Databricks (PySpark, Scala, SQL)
  • Orchestrate data workflows across ADF, Databricks, and Synapse Pipelines; implement modular and reusable data pipeline components
  • Develop advanced notebooks and production jobs in Azure Databricks (PySpark, SparkSQL, Delta Lake)
  • Optimize Spark jobs by tuning partitioning, caching, cluster configuration, and autoscaling for performance and cost
  • Implement Delta Lake for ACID-compliant data lakes and enable time travel/audit features
  • Engineer real-time data ingestion from Event Hubs, IoT Hub, and Kafka into Databricks and Synapse
  • Transform and enrich raw data, building robust data models and marts for analytics and AI use cases
  • Integrate structured, semi-structured, and unstructured sources, including APIs, logs, and files
  • Implement data validation, schema enforcement, and quality checks using Databricks, PySpark, and tools like Great Expectations
  • Manage access controls : Azure AD, Databricks workspace permissions, RBAC, Key Vault integration
  • Enable end-to-end lineage and cataloging via Microsoft Purview (or Unity Catalog if multi-cloud)
  • Automate deployment of Databricks assets (notebooks, jobs, clusters) using Databricks CLI/REST API, ARM/Bicep, or Terraform
  • Build and manage CI/CD pipelines in Azure DevOps for data pipelines and infrastructure as code
  • Containerize and deploy custom code with Azure Kubernetes Service (AKS) or Databricks Jobs as needed
  • Instrument monitoring and alerting with Azure Monitor, Log Analytics, and Databricks native tools
  • Diagnose and resolve performance bottlenecks in distributed Spark jobs and pipeline orchestrations
  • Collaborate with data scientists, BI engineers, and business stakeholders to deliver data solutions
  • Document design decisions, create technical specifications, and enforce engineering standards across the team

Required Skills & Experience

  • Hands-on with :
  • Azure Data Lake Gen2, Azure Data Factory, Azure Synapse Analytics, Azure Databricks
  • PySpark, SparkSQL, advanced SQL, Delta Lake
  • Data modeling (star/snowflake), partitioning, and data warehouse concepts
  • Strong Python programming and experience with workflow/orchestration (ADF, Airflow, or Synapse Pipelines)
  • Infrastructure automation : ARM/Bicep, Terraform, Databricks CLI/API, Azure DevOps
  • Deep understanding of Spark internals, cluster optimization, cost management, and distributed computing
  • Data security, RBAC, encryption, and compliance (SOC2, ISO, GDPR/DPDPA)
  • Excellent troubleshooting, performance tuning, and documentation skills

(ref:hirist.tech)

Skills

Data ValidationPythonData ModelingEtlSnowflakeAi/mlAnalyticsAiMlSqlData Engineer

If a job posting appears fraudulent, asks for payment, contains misleading information, or violates our guidelines, please report it immediately. Our team will review it promptly, Jobaaj does not charge any fee from the applicants.

Similar Jobs

View all
Loading...

Looking for something similar?

Bag Logo
Jobaaj
Get jobs like this by email

New Data Science Analytics & Machine Learning roles in Jaipur.

Job Alert
Google hiring for Specific Roles Apply Now!
1 min ago
New Opportunity
Amazon is hiring freshers Apply Now!
5 min ago
Featured Jobs
Microsoft opening 50+ positions Apply Now!
10 min ago

One email a day. Unsubscribe any time.