Azure Data Engineer

Azure Databricks (ADB) & PySpark

Responsibilities

• Develop and maintain data pipelines using Azure Databricks and PySpark.
• Design, build, and optimize ETL/ELT processes for large-scale data processing.
• Work with structured and unstructured data from multiple sources.
• Develop data transformation and validation logic using PySpark.
• Integrate Azure Databricks with Azure Data Factory and other Azure services.
• Optimize Spark jobs for performance and scalability.
• Troubleshoot and resolve data processing issues.
• Collaborate with business stakeholders, analysts, and data engineering teams.
• Follow best practices for coding, testing, and deployment.

Required Skills

• Strong hands-on experience with Azure Databricks and PySpark.
• Good knowledge of SQL and data warehousing concepts.
• Experience with Azure Data Factory (ADF).
• Understanding of Spark architecture and performance tuning.
• Experience working with Delta Lake and Data Lake concepts.
• Knowledge of Git and CI/CD processes.
• Strong analytical and problem-solving skills.

Good to Have

• Experience with Azure Synapse Analytics.
• Exposure to Power BI.
• Knowledge of Python and cloud-based data engineering practices.
• Experience in Banking, Finance, or Retail domains.