You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Kaxil Naik (Jira)" <ji...@apache.org> on 2020/03/31 01:52:00 UTC

[jira] [Updated] (AIRFLOW-6542) sparkKubernetes operator for https://github.com/GoogleCloudPlatform/spark-on-k8s-operator

     [ https://issues.apache.org/jira/browse/AIRFLOW-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kaxil Naik updated AIRFLOW-6542:
--------------------------------
    Fix Version/s:     (was: 1.10.10)
                   2.0.0

> sparkKubernetes operator for https://github.com/GoogleCloudPlatform/spark-on-k8s-operator
> -----------------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-6542
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6542
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: contrib
>    Affects Versions: 1.10.7
>            Reporter: Roi Teveth
>            Priority: Minor
>             Fix For: 2.0.0
>
>
> hi, 
> we  working on spark on Kubernetes POC using the google cloud platform spark-k8s-operator [https://github.com/GoogleCloudPlatform/spark-on-k8s-operator] and haven't found native airflow integration for it so we wrote one: 
> kubernetes_hook which create and get kuberenetes crd object
> spark_kubernetes_operator which sends sparkapplication crd to kubernetes cluster
> spark_kubernetes_sensor which poke sparkapplication state
> operator example with spark-pi application:[https://github.com/roitvt/airflow-spark-on-k8s-operator]
> I'll be glad to contribute our operator to airflow contrib
> Thanks
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)