Matheus Jericó
Menu
All projects

Project / 02

Spark on Kubernetes orchestration with Airflow

Run Spark jobs on Kubernetes from Airflow, with a custom operator, working deployment examples, and Google Cloud Storage integration.

Year
2024
Artifact type
Public source

What it demonstrates

The repository shows how an Airflow task can create and follow a Spark application on Kubernetes while keeping the operator, DAG, container setup, permissions, and example job visible in source.

Repository map

  • docker-airflow/custom_operator/spark_k8s_operator.py contains the custom operator.
  • docker-airflow/dag/spark_job_dag.py defines the example Airflow DAG.
  • docker-airflow/rbac/cluster-role.yaml and cluster-role-binding.yaml define Kubernetes access.
  • docker-custom-spark-operator/ and docker-custom-spark/ contain image definitions and Spark configuration.
  • example/wordcount-spark.yaml and example/wordcount.py provide the sample Spark application.

Inspect the source

Review the repository and the supporting article.