You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "Roi Teveth (Jira)" <ji...@apache.org> on 2020/01/12 15:29:00 UTC

[jira] [Created] (AIRFLOW-6542) sparkKubernetes operator for https://github.com/GoogleCloudPlatform/spark-on-k8s-operator

Roi Teveth created AIRFLOW-6542:
-----------------------------------

             Summary: sparkKubernetes operator for https://github.com/GoogleCloudPlatform/spark-on-k8s-operator
                 Key: AIRFLOW-6542
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6542
             Project: Apache Airflow
          Issue Type: Improvement
          Components: contrib
    Affects Versions: 1.10.7
            Reporter: Roi Teveth


hi, 

we  working on spark on Kubernetes POC using the google cloud platform spark-k8s-operator [https://github.com/GoogleCloudPlatform/spark-on-k8s-operator] and haven't found native airflow integration for it so we wrote one: 

kubernetes_hook which create and get kuberenetes crd object

spark_kubernetes_operator which sends sparkapplication crd to kubernetes cluster

spark_kubernetes_sensor which poke sparkapplication state

operator example with spark-pi application:[https://github.com/roitvt/airflow-spark-on-k8s-operator]

I'll be glad to contribute our operator to airflow contrib

Thanks

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)