[jira] [Commented] (AIRFLOW-6542) sparkKubernetes operator for https://github.com/GoogleCloudPlatform/spark-on-k8s-operator

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17245068#comment-17245068 ] 

ASF GitHub Bot commented on AIRFLOW-6542:
-----------------------------------------

sakshi-bansal commented on pull request #7163:
URL: https://github.com/apache/airflow/pull/7163#issuecomment-739774827


   Airflow version: 1.10.13
   
   Trying to deploy https://github.com/apache/airflow/blob/master/airflow/providers/cncf/kubernetes/example_dags/example_spark_kubernetes.py. Getting the following error **No module named 'airflow.providers'**. Executed **pip install apache-airflow-backport-providers-cncf-kubernetes** from the airflow worker/web/scheduler pod I can now import the `from airflow.providers.cncf.kubernetes.operators.spark_kubernetes import SparkKubernetesOperator`  . Getting the following error :
   Traceback (most recent call last):
     File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/dagbag.py", line 256, in process_file
       m = imp.load_source(mod_name, filepath)
     File "/usr/local/lib/python3.6/imp.py", line 172, in load_source
       module = _load(spec)
     File "<frozen importlib._bootstrap>", line 684, in _load
     File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
     File "<frozen importlib._bootstrap_external>", line 678, in exec_module
     File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
     File "/opt/airflow/dags/sparkoperator.py", line 4, in <module>
       from airflow.providers.cncf.kubernetes.operators.spark_kubernetes import SparkKubernetesOperator
   ModuleNotFoundError: No module named 'airflow.providers'
   /home/airflow/.local/lib/python3.6/site-packages/airflow/kubernetes/pod_generator.py:39: DeprecationWarning: This module is deprecated. Please use `airflow.kubernetes.pod`.
     from airflow.contrib.kubernetes.pod import _extract_volume_mounts 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> sparkKubernetes operator for https://github.com/GoogleCloudPlatform/spark-on-k8s-operator
> -----------------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-6542
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6542
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: contrib
>    Affects Versions: 1.10.7
>            Reporter: Roi Teveth
>            Priority: Minor
>             Fix For: 2.0.0
>
>
> hi, 
> we  working on spark on Kubernetes POC using the google cloud platform spark-k8s-operator [https://github.com/GoogleCloudPlatform/spark-on-k8s-operator] and haven't found native airflow integration for it so we wrote one: 
> kubernetes_hook which create and get kuberenetes crd object
> spark_kubernetes_operator which sends sparkapplication crd to kubernetes cluster
> spark_kubernetes_sensor which poke sparkapplication state
> operator example with spark-pi application:[https://github.com/roitvt/airflow-spark-on-k8s-operator]
> I'll be glad to contribute our operator to airflow contrib
> Thanks
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)