You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Chris (Jira)" <ji...@apache.org> on 2021/03/31 19:42:00 UTC
[jira] [Commented] (AIRFLOW-2535) TriggerDagRunOperator ignores
max_active_runs, leading to scheduler breakdown
[ https://issues.apache.org/jira/browse/AIRFLOW-2535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17312673#comment-17312673 ]
Chris commented on AIRFLOW-2535:
--------------------------------
This is making me avoid the TriggerDagRunOperator like the plague. I think I'll stick with Sensors until this gets sorted out.
> TriggerDagRunOperator ignores max_active_runs, leading to scheduler breakdown
> -----------------------------------------------------------------------------
>
> Key: AIRFLOW-2535
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2535
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators, scheduler
> Affects Versions: 1.9.0
> Environment: CeleryExec, Rabbitmq, mysql
> Reporter: Tylar Murray
> Priority: Minor
>
> `TriggerDagRunOperator` does not respect the `max_active_runs` setting. This in itself is not a huge issue, but the scheduler breaks down if `max_active_runs` is exceeded. Task scheduling becomes absurdly slow, quickly leading to perpetual DAG buildup.
> `TriggerDagRunOperator` could throw exception if `max_active_runs` will be exceeded, or maybe dags could be put into a `queued` state rather than directly into `running`?
> Alternatively: the scheduler probably shouldn't slow task scheduling when `max_active_runs` is exceeded.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)