Job Description
### Role Summary
We are looking for an experienced **GCP Data Engineer** to design and build an automated solution for **data discovery, end-to-end lineage, metadata management, and data quality** across multiple GCP projects. The solution will trace data movement from source systems through GCP processing layers, store metadata centrally in BigQuery, and integrate lineage information with **Collibra using OpenLineage APIs**.
### Key Responsibilities
- Build programmatic **upstream and downstream data lineage** across multiple GCP projects and BigQuery datasets.
- Discover datasets, tables, views, stored procedures, jobs, and other relevant data assets.
- Trace data movement and intermediate processing through **BigQuery, Dataflow, Cloud Data Fusion, stored procedures, Cloud Storage, and related GCP services**.
- Use **GCP Data Lineage/Dataplex APIs, BigQuery INFORMATION_SCHEMA, job metadata, and other GCP APIs** to collect and enrich lineage information.
- Capture available **table-level and column-level lineage**, transformation details, and pipeline execution metadata.
- Extract **data quality rules, scans, and results** using Dataplex and related metadata services.
- Develop **Python/SQL-based services and pipelines** to collect, normalize, validate, and store metadata in BigQuery.
- Generate standardized **OpenLineage JSON/events** and integrate lineage metadata with **Collibra**.
- Implement secure **cross-project access using GCP service accounts, IAM roles, and least-privilege principles**.
- Build monitoring, logging, error handling, and audit capabilities for metadata and lineage processing.
- Support **AI/agentic workflows** for metadata analysis, lineage enrichment, and identification of lineage or data quality gaps.
### Required Skills
- Strong hands-on experience with **Python and SQL**
- Strong experience with **Google BigQuery**, including metadata and `INFORMATION_SCHEMA`
- Experience with **Dataflow and/or Cloud Data Fusion**
- Experience integrating with **GCP APIs and REST APIs**
- Understanding of **data lineage, metadata management, data quality, and data governance**
- Experience with **GCP IAM, service accounts, and cross-project access**
- Experience designing and supporting production-grade data pipelines
- Strong troubleshooting and problem-solving skills
### Preferred Skills
- **Dataplex / GCP Data Lineage API**
- **Collibra and OpenLineage**
- Cloud Run and Cloud Storage
- Terraform / Infrastructure as Code
- Vertex AI / agentic or LLM-based workflows
- SQL parsing and transformation dependency analysis
- Enterprise data governance and metadata platforms
### Experience
- **5+ years of data engineering experience**
- Strong hands-on experience with **GCP data services**
- Experience working in **enterprise, multi-project cloud environments**
- Ability to collaborate with data engineering, architecture, cloud security, and data governance teams
### Key Deliverables
The engineer will help deliver a scalable solution for **cross-project data discovery, end-to-end lineage, data quality metadata, centralized BigQuery metadata storage, and Collibra/OpenLineage integration**.
