Databricks Engineer (Remote)

• Databricks Engineer (Remote)
• Company: HireFlex
• Job Location: Remote (Work from home), India
• Employment Type: Full-time
• Experience Required: 7+ years
• Role Overview

We are looking for a Data Engineer with strong experience in Azure data engineering, Databricks and data lake technologies, along with an understanding of IT Security/Cybersecurity use cases. The ideal candidate is a generalist who can work across security data engineering, analytics, Data Science and emerging GenAI use cases.

• Key Responsibilities
• Design, develop and maintain scalable data pipelines for IT Security and Cybersecurity use cases.
• Build and manage data solutions using Azure Data Lake Storage (ADLS/ADLS Gen2), Azure Databricks and Azure data services.
• Ingest, transform and process security data from multiple sources such as SIEM, endpoint, identity, network, cloud and application security platforms.
• Develop Databricks/PySpark solutions for processing and analysing large volumes of security telemetry.
• Work with security logs, alerts, vulnerability data, threat intelligence, authentication data and other security datasets.
• Work with Azure Security services and understand security-related data generated across the Azure/Microsoft security ecosystem.
• Develop data models and curated datasets to support Security Operations, Threat Detection, Risk Analytics and reporting.
• Collaborate with Data Scientists to prepare datasets for machine learning, anomaly detection, behavioural analytics and predictive use cases.
• Support GenAI/LLM/RAG use cases involving security data, knowledge retrieval and intelligent security analytics.
• Develop automation scripts using Python Build and maintain Power BI dashboards and reports for security metrics, trends, risks and operational KPIs.

Implement data quality, governance, access controls and monitoring across data pipelines. Troubleshoot data ingestion, transformation and pipeline issues and optimize performance and cost.

• Collaborate with Cybersecurity, SOC, Data Engineering, Data Science and Cloud Engineering teams.
• Required Skills & Experience
• Senior Data Engineer with proven experience delivering complex data and analytics solutions in enterprise environments.

Strong hands-on experience with Data Science, LLMs, Agentic AI frameworks, MCP, and RAG architectures, including designing, building, implementing, and troubleshooting solutions in production.

Minimum 7 years of hands-on Databricks experience, including architecture, development, optimization, operations, and support of large-scale data platforms.

Demonstrated experience working with cybersecurity data and analytics, translating security requirements into actionable insights, detection capabilities, and reporting.

Hands-on experience with Microsoft security platforms, particularly Microsoft Sentinel, Microsoft Defender, and Azure Data Explorer (ADX), including data ingestion, analytics, investigations, and operational use cases.

Proven ability to lead initiatives end-to-end, from requirements gathering and solution design through implementation, deployment, and operationalization.

Ability to mentor and guide junior engineers, providing technical leadership, coaching, and best-practice recommendations.

Responsible for performing quality assurance and technical reviews of own work and the work of junior Data Engineers/Data Scientists, ensuring coding standards, solution quality, documentation, and operational readiness.

• Ability to provide concrete examples of delivered solutions, technical challenges solved, architecture decisions made, and measurable business or security outcomes.
• Strong ownership mindset with a track record of personally driving implementation, not only coordinating or overseeing delivery.
• Data Science / AI Exposure
• Good understanding of Data Science concepts, including statistics, machine learning, anomaly detection and feature engineering.
• Ability to work with Data Scientists and understand ML/data requirements. Exposure to GenAI, LLM, RAG or AI-powered analytics is preferred.
• Ability to explore and analyse large security datasets and identify meaningful patterns and trends.
• Preferred Certifications:
• Databricks Certified Data Engineer Associate / Professional or equivalent Databricks certification.

Microsoft security certifications such as SC-200 Security Operations Analyst Associate, SC-500 Cloud and AI Security Engineer AssociateOther relevant certifications such as Security+, CEH, CISSP or equivalent are an advantage. .