[jira] [Updated] (AIRFLOW-3857) spark_submit_hook cannot kill pod in kubernetes

["Aizhamal Nurmamat kyzy (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aizhamal Nurmamat kyzy updated AIRFLOW-3857:
--------------------------------------------
    Component/s:     (was: kubernetes)

> spark_submit_hook cannot kill pod in kubernetes
> -----------------------------------------------
>
>                 Key: AIRFLOW-3857
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3857
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: contrib
>    Affects Versions: 1.10.0, 1.10.1, 1.10.2
>         Environment: any
>            Reporter: Olivier Girardot
>            Assignee: Olivier Girardot
>            Priority: Major
>             Fix For: 2.0.0
>
>
> When trying to kill a driver pod (for example due to a timeout) the `spark_submit_hook` does not use the proper kubernetes api and ends up calling a non-existent member on the CoreV1Api object : 
>  
> {code:java}
> [2019-02-09 07:06:19,883] {{logging_mixin.py:95}} INFO - [2019-02-09 07:06:19,883] {{spark_submit_hook.py:552}} INFO - Killing pod ...-20190209-a48f77e00aa0367ca39f6495b8f6cc08-driver on Kubernetes
> [2019-02-09 07:06:19,892] {{models.py:1788}} ERROR - 'CoreV1Api' object has no attribute 'V1DeleteOptions'
> Traceback (most recent call last):
>  File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 1652, in _run_raw_task
>  result = task_copy.execute(context=context)
> ...
>  File "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/spark_submit_hook.py", line 332, in submit
>  self._process_spark_submit_log(iter(self._submit_sp.stdout.readline, ''))
>  File "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/spark_submit_hook.py", line 379, in _process_spark_submit_log
>  for line in itr:
>  File "/usr/local/lib/python3.6/site-packages/airflow/utils/timeout.py", line 42, in handle_timeout
>  raise AirflowTaskTimeout(self.error_message)
> airflow.exceptions.AirflowTaskTimeout: Timeout
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
>  File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 1654, in _run_raw_task
>  task_copy.on_kill()
>  ...
>  File "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/spark_submit_hook.py", line 560, in on_kill
>  body=client.V1DeleteOptions(),
> AttributeError: 'CoreV1Api' object has no attribute 'V1DeleteOptions'
> [2019-02-09 07:06:19,896] {{models.py:1817}} INFO - All retries failed; marking task as FAILED
> {code}
>  
> This is only due to the use of the kube_client (which is really the kubernetes.client.CoreV1Api) instead of the kubernetes.client.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)