Leading the build of a data mesh platform
Founding-team technical leadership for a data mesh product: the core platform, its APIs, compute orchestration, data sharing and release management.
Challenge
A product company set out to build a data mesh platform from nothing: a core data platform and APIs that give teams domain-oriented access to data, across more than one compute engine and more than one cloud.
It needed the platform built and, as the team grew, someone to own architecture, technical strategy and delivery across several services.
What we did
- Designed and built the core data platform and the REST APIs behind the data mesh, giving domain-oriented data access across teams.
- Built a compute orchestration layer that runs batch and streaming workloads on Kubernetes, Apache Spark, Databricks and Amazon EMR.
- Added data sharing in both directions without copying data: inbound virtualisation of Apache Iceberg tables through Trino, and an outbound open-protocol provider that serves Iceberg tables to any compliant client.
- Introduced an MLOps framework on MLflow that standardises experimentation, model tracking and deployment.
- Moved service secrets from PostgreSQL to HashiCorp Vault with a dual-stack rollout.
- Led the application team: architecture decisions, release management on a weekly cadence, coaching and regular feedback.
- Set data governance principles for the mesh, formalising domain ownership.
Outcome
- The platform is in production and gives teams data access by domain.
- Data is shared with other platforms in both directions without duplication.
- The secrets migration completed with zero downtime.
- Weekly production releases across the application services stayed stable.
Technologies
Kubernetes, Apache Spark, Databricks, Amazon EMR, Trino, Apache Iceberg, MLflow, HashiCorp Vault, PostgreSQL