You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Daniel Imberman (Jira)" <ji...@apache.org> on 2020/02/06 17:35:00 UTC
[jira] [Commented] (AIRFLOW-6742) Task instance state set to failed
even though the Pod succeeded when using KubernetesExecutor
[ https://issues.apache.org/jira/browse/AIRFLOW-6742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17031791#comment-17031791 ]
Daniel Imberman commented on AIRFLOW-6742:
------------------------------------------
Hi Andrew,
Can you post the logs from the pod launched from the KubernetesPodOperator? I'm not seeing anything in the scheduler logs that suggest that the kubernetesexecutor set the task to failed.
> Task instance state set to failed even though the Pod succeeded when using KubernetesExecutor
> ---------------------------------------------------------------------------------------------
>
> Key: AIRFLOW-6742
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6742
> Project: Apache Airflow
> Issue Type: Bug
> Components: executor-kubernetes
> Affects Versions: 1.10.7, 1.10.8
> Environment: Kubernetes (1.16), Airflow (1.10.7, 10.10.8rc1)
> Reporter: Andrew Cleland
> Assignee: Daniel Imberman
> Priority: Major
> Labels: State, executor, kubernetes, taskinstance
> Attachments: airflow_scheduler_logs.txt, failed_dag_run.png, failed_task_instance.png, k8s_pods.png, kubernetes_executor_logs.txt
>
>
> When running a KubernetesPodOperator task with the KubernetesExecutor, the Pod succeeds but Airflow sets the task instance state to Failed.
> Attached files:
> * k8s_pods.png - KubernetesExecutor pod and KubernetesPodOperator pod both succeeded
> * kubernetes_executor_logs - Launched the KubernetesPodOperator successfully
> * airflow_scheduler_logs - "Found matching task with current state failed"
> * failed_task_instance - The failed task instance in the airflow UI
> * failed_dag_run - The failed dag run in the airflow UI
> It seems that the database is being updated with task state of failed, but I'm not sure whereabouts this state is being changed. [Here|https://github.com/apache/airflow/blob/1.10.7/airflow/contrib/executors/kubernetes_executor.py#L628] is the line where the KubernetesExecutor queries the database and finds a failed task.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)