You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Anand (JIRA)" <ji...@apache.org> on 2019/07/01 09:30:00 UTC
[jira] [Comment Edited] (AIRFLOW-4855) Task remains in queued state
[ https://issues.apache.org/jira/browse/AIRFLOW-4855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16876049#comment-16876049 ]
Anand edited comment on AIRFLOW-4855 at 7/1/19 9:29 AM:
--------------------------------------------------------
[~ash], I was using K8s executor and observed that if there is a failure in the following method- [https://github.com/apache/airflow/blob/65eef1ca3d848ecf02eb79c41a72d736e40a9108/airflow/models/taskinstance.py#L836,]
where TI increase the try_number and mark the task status to RUNNING, the tasks are never retried and remain in QUEUED state.
executor gets a failed event from Kubewatcher and it is processed here- [https://github.com/apache/airflow/blob/65eef1ca3d848ecf02eb79c41a72d736e40a9108/airflow/jobs/scheduler_job.py#L1233]
but since the try_number was not incremented by the task, this condition never meets- [https://github.com/apache/airflow/blob/65eef1ca3d848ecf02eb79c41a72d736e40a9108/airflow/jobs/scheduler_job.py#L1243]
and the tasks are not even marked for retry even if they have retries configured. I think the behaviour should be same for other executors and also I think this is a rare scenario, but I still did encounter it.
I am proposing this change,where in case of this failure, I will mark the task state as NONE, so that the are picked up again for running.
PR link- [https://github.com/apache/airflow/pull/5506/files]
was (Author: anand.g14):
[~ash], I was using K8s executor and observed that if there is a failure in the following method- [https://github.com/apache/airflow/blob/65eef1ca3d848ecf02eb79c41a72d736e40a9108/airflow/models/taskinstance.py#L836,]
where TI increase the try_number and mark the task status to RUNNING, the tasks are never retried.
executor gets a failed event from Kubewatcher and it is processed here- [https://github.com/apache/airflow/blob/65eef1ca3d848ecf02eb79c41a72d736e40a9108/airflow/jobs/scheduler_job.py#L1233]
but since the try_number was not incremented by the task, this condition never meets- [https://github.com/apache/airflow/blob/65eef1ca3d848ecf02eb79c41a72d736e40a9108/airflow/jobs/scheduler_job.py#L1243]
and the tasks are not even marked for retry even if they have retries configured. I think the behaviour should be same for other executors and also I think this is a rare scenario, but I still did encounter it.
I am proposing this change,where in case of this failure, I will mark the task state as NONE, so that the are picked up again for running.
PR link- [https://github.com/apache/airflow/pull/5506/files]
> Task remains in queued state
> ----------------------------
>
> Key: AIRFLOW-4855
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4855
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 1.10.3
> Reporter: Anand
> Assignee: Anand
> Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)