You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "jack (Jira)" <ji...@apache.org> on 2019/10/28 22:50:00 UTC

[jira] [Commented] (AIRFLOW-5286) Add requeue logic to airflow scheduler and executor

    [ https://issues.apache.org/jira/browse/AIRFLOW-5286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961526#comment-16961526 ] 

jack commented on AIRFLOW-5286:
-------------------------------

Sounds useful! Are you planing to PR your code?

> Add requeue logic to airflow scheduler and executor
> ---------------------------------------------------
>
>                 Key: AIRFLOW-5286
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5286
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: executors, scheduler
>    Affects Versions: 1.10.4
>            Reporter: Yingbo Wang
>            Assignee: Yingbo Wang
>            Priority: Major
>
> Airflow queued tasks sometime stuck for long time without being picked up. In many cases the root cause is hard to debug. 
> The proposed solution here is to add requeue logic in airflow scheduler and executor. For tasks with a queued state in metaDB. If the task did not show in executor.queued_tasks in a certain inteval, we requeue the task so that stuck tasks can be released and picked up. The solution was used in Airbnb and proven to be helpful. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)