You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "Tylar Murray (JIRA)" <ji...@apache.org> on 2018/05/29 13:46:00 UTC

[jira] [Created] (AIRFLOW-2535) TriggerDagRunOperator ignores max_active_runs, leading to scheduler breakdown

Tylar Murray created AIRFLOW-2535:
-------------------------------------

             Summary: TriggerDagRunOperator ignores max_active_runs, leading to scheduler breakdown
                 Key: AIRFLOW-2535
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2535
             Project: Apache Airflow
          Issue Type: Bug
          Components: operators, scheduler
    Affects Versions: 1.9.0
         Environment: CeleryExec, Rabbitmq, mysql
            Reporter: Tylar Murray


`TriggerDagRunOperator` does not respect the `max_active_runs` setting. This in itself is not a huge issue, but the scheduler breaks down if `max_active_runs` is exceeded. Task scheduling becomes absurdly slow quickly leading to perpetual DAG buildup. 

`TriggerDagRunOperator` could throw exception if `max_active_runs` will be exceeded, or maybe dags could be put into a `queued` state rather than directly into `running`?

Alternatively: the scheduler probably shouldn't slow task scheduling when `max_active_runs` is exceeded.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)