Senior Databrics Architect

Role & responsibilities

Architecture & Design

• Define and own the target lakehouse architecture on Azure Databricks, including medallion (bronze / silver / gold) layering, storage design on ADLS Gen2, and integration patterns with Azure Data Factory.
• Design canonical and dimensional data models for finance, pricing, and operational domains, and set modeling standards the whole team follows.
• Produce architecture decision records (ADRs) so that platform choices are documented, reviewable, and reproducible.
• Design ingestion patterns for diverse source systems including AS400 / DB2, Oracle, SQL Server, transportation management systems, and SaaS finance applications.

Data Governance with Unity Catalog

• Design and implement the Unity Catalog structure (metastores, catalogs, schemas) aligned to environments and business domains
• . Implement fine-grained access control, row- and column-level security, and data masking for sensitive data.
• Establish data ownership and stewardship, lineage, tagging, and audit practices using Unity Catalog capabilities. Forward Air | Omni Logistics
• Senior Databricks Architect
• Partner with Finance, Information Security, and business data owners to translate governance policies into enforceable platform controls.
• Define data quality rules and expectations (e.g., Delta Live Tables expectations or equivalent) and embed them into pipelines.

Hands-On Engineering

• Build and review PySpark, Spark SQL, and Delta Lake code; lead by example on code quality, testing, and documentation.
• Design orchestration using Databricks Workflows and ADF with clear dependencies, retries, and failure handling.
• Tune performance and cost: cluster policies, job compute vs. all-purpose compute, Photon, partitioning, Z-ordering / liquid clustering, and table maintenance.
• Troubleshoot complex production issues and drive root-cause fixes rather than workarounds.

Migration & Modernization

• Plan and lead the migration of workloads from legacy warehouses and semantic layers (e.g., Redshift, SSAS) onto Databricks, including validation and cutover.
• Rationalize redundant pipelines and tools into a simpler, consistent platform.

Leadership & Collaboration

• Act as the technical authority for the data platform across US and India teams; run design reviews and mentor engineers.
• Work with DevOps to ensure every pipeline is deployed through CI/CD with environment parity across Dev, QA, and Production.
• Communicate architecture choices and trade-offs clearly to technical and non-technical stakeholders.

Preferred candidate profile

• 12-15 years in data engineering and data architecture, with 4+ years of hands-on Azure Databricks in production.
• Deep, practical experience implementing Unity Catalog: catalog design, access control, lineage, and governance policies.
• Expert in PySpark, Spark SQL, and Delta Lake, including performance tuning at scale.
• Strong data modeling skills (dimensional, data vault, or canonical) for enterprise reporting and analytics.
• Experience with Azure Data Factory, ADLS Gen2, Azure Key Vault, and Entra ID integration.
• Proven record of migrating workloads from legacy data warehouses to a lakehouse.
• Working knowledge of CI/CD for data (Azure DevOps or GitHub), Git branching, and Databricks Asset Bundles.
• Excellent written and verbal communication; comfortable presenting to senior leadership.