You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "Yingbo Wang (Jira)" <ji...@apache.org> on 2019/08/22 22:01:00 UTC

[jira] [Created] (AIRFLOW-5286) Add requeue logic to airflow scheduler and executor

Yingbo Wang created AIRFLOW-5286:
------------------------------------

             Summary: Add requeue logic to airflow scheduler and executor
                 Key: AIRFLOW-5286
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5286
             Project: Apache Airflow
          Issue Type: Improvement
          Components: executors, scheduler
    Affects Versions: 1.10.4
            Reporter: Yingbo Wang
            Assignee: Yingbo Wang


Airflow queued tasks sometime stuck for long time without being picked up. In many cases the root cause is hard to debug. 

The proposed solution here is to add requeue logic in airflow scheduler and executor. For tasks with a queued state in metaDB. If the task did not show in executor.queued_tasks in a certain inteval, we requeue the task so that stuck tasks can be released and picked up. The solution was used in Airbnb and proven to be helpful. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)