Data services and pipelines for maritime defence
Python services, REST APIs and batch pipelines implementing a domain data model, integrated with the organisation's data mesh platform.
Challenge
A defence organisation working in the maritime domain needed its data model implemented as working services, and the data it collects persisted and turned into data products.
The work had to fit the organisation's existing data mesh platform and its formal processes.
What we did
- Built Python services and REST APIs with FastAPI and Pydantic that implement the domain data model.
- Built batch pipelines on Apache Airflow for data persistence and data product generation, integrated with the organisation's data mesh platform.
- Used MongoDB and SeaweedFS for storage, and open geospatial data formats.
- Delivered full-stack features in React, TypeScript and Streamlit as part of the organisation's engineering team.
- Automated deployments with Docker Compose and GitLab CI/CD.
Outcome
- The data model runs as services with documented APIs.
- Collected data is persisted and published as data products on the organisation's platform.
- Deployments are automated.
Technologies
Python, FastAPI, Pydantic, Apache Airflow, MongoDB, SeaweedFS, React, TypeScript, Docker, GitLab CI/CD