You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Aizhamal Nurmamat kyzy (JIRA)" <ji...@apache.org> on 2019/05/18 02:46:02 UTC
[jira] [Updated] (AIRFLOW-3879) When a pod created by
KubernetesPodOperator is evicted the task fails
[ https://issues.apache.org/jira/browse/AIRFLOW-3879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aizhamal Nurmamat kyzy updated AIRFLOW-3879:
--------------------------------------------
Labels: kubernetes (was: )
Component/s: (was: kubernetes)
operators
Moving to operators component, and labeling with kubernetes as part of the component refactor.
> When a pod created by KubernetesPodOperator is evicted the task fails
> ---------------------------------------------------------------------
>
> Key: AIRFLOW-3879
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3879
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
> Affects Versions: 1.10.0
> Reporter: Jane
> Priority: Major
> Labels: kubernetes
>
> When a pod is evicted it's status does not match the expected status, causing the task to error due to a Type error.
>
> At a minimum the status of the pod should be reported in the error.
> {code:java}
> [2019-02-07 18:56:19,058] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique [2019-02-07 18:56:19,055] {models.py:1736} ERROR - 'NoneType' object is not iterable
> [2019-02-07 18:56:19,058] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique Traceback (most recent call last):
> [2019-02-07 18:56:19,059] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique File "/usr/local/lib/airflow/airflow/models.py", line 1633, in _run_raw_task
> [2019-02-07 18:56:19,059] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique result = task_copy.execute(context=context)
> [2019-02-07 18:56:19,059] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique File "/home/airflow/gcs/dags/workflow/operators/airflow_operators.py", line 112, in execute
> [2019-02-07 18:56:19,059] {models.py:1764} INFO - Marking task as FAILED.
> [2019-02-07 18:56:19,059] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique return super().execute(context)
> [2019-02-07 18:56:19,059] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique File "/usr/local/lib/airflow/airflow/contrib/operators/kubernetes_pod_operator.py", line 115, in execute
> [2019-02-07 18:56:19,059] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique get_logs=self.get_logs)
> [2019-02-07 18:56:19,059] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique File "/usr/local/lib/airflow/airflow/contrib/kubernetes/pod_launcher.py", line 81, in run_pod
> [2019-02-07 18:56:19,059] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique return self._monitor_pod(pod, get_logs)
> [2019-02-07 18:56:19,059] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique File "/usr/local/lib/airflow/airflow/contrib/kubernetes/pod_launcher.py", line 98, in _monitor_pod
> [2019-02-07 18:56:19,059] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique while self.base_container_is_running(pod):
> [2019-02-07 18:56:19,059] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique File "/usr/local/lib/airflow/airflow/contrib/kubernetes/pod_launcher.py", line 127, in base_container_is_running
> [2019-02-07 18:56:19,060] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique event.status.container_statuses)), None)
> [2019-02-07 18:56:19,060] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique TypeError: 'NoneType' object is not iterable
> [2019-02-07 18:56:19,060] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique [2019-02-07 18:56:19,059] {models.py:1764} INFO - Marking task as FAILED.
> [2019-02-07 18:56:20,512] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique Traceback (most recent call last):
> [2019-02-07 18:56:20,512] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique File "/usr/local/bin/airflow", line 6, in <module>
> [2019-02-07 18:56:20,512] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique exec(compile(open(__file__).read(), __file__, 'exec'))
> [2019-02-07 18:56:20,512] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique File "/usr/local/lib/airflow/airflow/bin/airflow", line 32, in <module>
> [2019-02-07 18:56:20,512] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique args.func(args)
> [2019-02-07 18:56:20,512] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique File "/usr/local/lib/airflow/airflow/utils/cli.py", line 74, in wrapper
> [2019-02-07 18:56:20,512] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique return f(*args, **kwargs)
> [2019-02-07 18:56:20,513] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique File "/usr/local/lib/airflow/airflow/bin/cli.py", line 496, in run
> [2019-02-07 18:56:20,513] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique _run(args, dag, ti)
> [2019-02-07 18:56:20,513] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique File "/usr/local/lib/airflow/airflow/bin/cli.py", line 402, in _run
> [2019-02-07 18:56:20,513] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique pool=args.pool,
> [2019-02-07 18:56:20,513] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique File "/usr/local/lib/airflow/airflow/utils/db.py", line 74, in wrapper
> [2019-02-07 18:56:20,513] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique return func(*args, **kwargs)
> [2019-02-07 18:56:20,513] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique File "/usr/local/lib/airflow/airflow/models.py", line 1633, in _run_raw_task
> [2019-02-07 18:56:20,513] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique result = task_copy.execute(context=context)
> [2019-02-07 18:56:20,513] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique File "/home/airflow/gcs/dags/workflow/operators/airflow_operators.py", line 112, in execute
> [2019-02-07 18:56:20,513] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique return super().execute(context)
> [2019-02-07 18:56:20,513] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique File "/usr/local/lib/airflow/airflow/contrib/operators/kubernetes_pod_operator.py", line 115, in execute
> [2019-02-07 18:56:20,513] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique get_logs=self.get_logs)
> [2019-02-07 18:56:20,513] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique File "/usr/local/lib/airflow/airflow/contrib/kubernetes/pod_launcher.py", line 81, in run_pod
> [2019-02-07 18:56:20,514] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique return self._monitor_pod(pod, get_logs)
> [2019-02-07 18:56:20,514] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique File "/usr/local/lib/airflow/airflow/contrib/kubernetes/pod_launcher.py", line 98, in _monitor_pod
> [2019-02-07 18:56:20,514] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique while self.base_container_is_running(pod):
> [2019-02-07 18:56:20,514] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique File "/usr/local/lib/airflow/airflow/contrib/kubernetes/pod_launcher.py", line 127, in base_container_is_running
> [2019-02-07 18:56:20,514] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique event.status.container_statuses)), None)
> [2019-02-07 18:56:20,514] {base_task_runner.py:107} INFO - Job 10690: Subtask data-processing-unique TypeError: 'NoneType' object is not iterable
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)