Scaling a data platform on Kubernetes

Problem

The platform needed a reliable way to move a large Airflow estate from Dataproc jobs to Spark applications on Kubernetes. That migration sat within a broader engineering problem: connecting database changes, event streams, batch loads, transformations, and interactive exploration through a shared data platform.

My contribution

I designed and implemented the Spark execution environment (Minecraft), its Airflow integration (Minecraft Operator), and the CDC abstraction (Chupa Cabra). I also co-designed and led the shared Spark tooling (Sness-lib) and declarative DAG generator (Witcher).

System

• Sources and ingestion: Chupa Cabra abstracted database change data capture (CDC) over Kafka Connect and Debezium. The ingestion layer also consumed events from Pub/Sub, while some applications published directly to Kafka.
• Batch processing: Witcher generated Airflow DAGs from YAML for full and incremental loads. Spark jobs read directly from databases through JDBC or processed CDC from Kafka, with checkpointing and deduplication.
• Streaming: PySpark workloads on Kubernetes consumed Kafka events and loaded data into the lake and analytical storage.
• Storage and analytics: Delta Lake on Google Cloud Storage supported the lakehouse, alongside BigQuery for analytical consumption and dbt transformations.
• Developer workflows: Sness-lib standardized Spark workloads and Data Lake access. Users authored transformation pipelines and worked with PySpark in JupyterHub exploration environments.
• Execution and infrastructure: Minecraft provided the Kubernetes environment with Spark Operator. Minecraft Operator created, tracked, and monitored Spark applications from Airflow. Terraform provisioned the GCP infrastructure.

My implementation and technical leadership responsibilities are described above. The wider platform included products and integrations delivered and operated by the platform teams.

Outcome

Equivalent workloads moved from Dataproc to Spark on Kubernetes through a staged migration, improving infrastructure efficiency while preserving the shared ingestion, orchestration, and analytical workflows used by the platform teams.

Team credit

Delivered with the data platform teams. I set technical direction across multiple platform teams; implementation and day-to-day operation were shared responsibilities.

Architecture proposal: platform assistant

Status: architecture proposal documented. Implementation prerequisites remained pending.

I proposed a conversational assistant to help platform users find datasets, understand lineage and ownership, and investigate job execution status. The design connected catalog, documentation, and operational evidence through a single conversation.

• Access: Google Chat or a chatbot embedded in the platform console, with an ingress layer and Pub/Sub connecting requests to the agent.
• Agent and tools: A Pi-based harness with runbooks and workflows, using MCP connectors for OpenMetadata, Confluence, and Spark History Server, plus tools for BigQuery, GCS, Kubernetes, logging, and monitoring. Documentation would be retrieved when a question was asked.
• Operating boundaries: Read-only tools, scoped permissions, auditable calls, and escalation when the available evidence could not answer a question.

The proposal identified prerequisites: complete catalog and lineage ingestion, update the platform documentation, and deploy and configure Spark History Server with event logs. Pilot scope, channel rollout, identity handling, and runtime choices were still open. Production results and adoption metrics have not been established for this assistant.

Explore the assistant architecture proposal for its workflow, prerequisites, and open decisions.