Case study / 05
Building an Airflow-driven ML platform on Kubernetes
Orchestrating acquisition, training, validation, and API integration for a Kubernetes-based machine learning platform.
- Area
- Machine learning infrastructure
- Role
- Kubernetes Developer
System architecture
Machine learning workflow
An API-driven Airflow control plane coordinates acquisition, training, validation, and model evidence on Kubernetes.
Kubernetes
- 01Acquire
- 02Train
- 03Validate
Context
This Kubernetes-based machine learning platform coordinated the work needed to acquire data, train models, validate models, and integrate the workflow through an API.
My contribution
I created the Airflow DAGs for data acquisition, model training, and model validation, and built the API layer that interacted with Airflow.
Platform flow
The platform flow was acquisition -> training -> validation -> API-driven orchestration. Helm and CI/CD supported the platform’s Airflow, API, and UI components.
Team credit
I worked within the client’s engineering team, with responsibility for the Airflow workflows and their API integration.
System materials
Stack
- Apache Airflow
- Kubernetes
- MLflow
- Python
- Helm
- Azure