[GitHub] [airflow] akotuc opened a new issue #17375: Scheduler "deadlocks" itself when max_active_runs_per_dag is reached by restart of failed task in previous run

[GitBox <gi...@apache.org>]

akotuc opened a new issue #17375:
URL: https://github.com/apache/airflow/issues/17375


   _**Apache Airflow version: 2.1.2**_
   
   **What happened:**
   Let's say we have DAG with max_active_runs = 1 and wait_for_downstream = true config with hourly schedule. Now one of the task fails and DAG run change its status to failed. Next DAG hourly run triggers after some time however tasks don't run due to wait_for_downstream dependency. The issue why the previous DAG run failed is resolved and the failed tasks are cleared - now we have 2 DAG runs in state active. The newer DAG run tasks wait on downstream dependency and the previous DAG run tasks are not scheduled due to 1 active DAG runs limit. We have kind of deadlock since neither of active DAG runs actually schedule tasks.
   
   _Note:_ might be related to https://github.com/apache/airflow/issues/14205
   
   _Executor:_ CeleryExecutor
   
   **What you expected to happen:**
   Would expect the DAg run with older execution date starts trigger the tasks since newer DAG run is waiting on downstream dependency.
   
   **How to reproduce it:**
   - DAG with hourly schedule, tasks A,B,C with dependency A -> B -> C and config max_active_runs = 1 and wait_for_downstream = true
   - Make sure task A fails in DAG run with execution date e.g. 06:00:00 -> DAG run state changes to failed
   - Let another hourly DAG run (e.g. 07:00:00) start -> DAG run state is active however tasks are not scheduled due to wait on downstream dependency
   - Clear failed tasks in 06:00:00 DAG run:
      * now we have 2 active DAG runs (06:00:00 and 07:00:00)
      * tasks in 07:00:00 are not scheduled due to wait on downstream dependency
      * tasks in 06:00:00 are not scheduled since it 2nd active DAG but only 1 active DAG is allowed


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] nikie commented on issue #17375: Scheduler "deadlocks" itself when max_active_runs_per_dag is reached by restart of failed task in previous run

[GitBox <gi...@apache.org>]

nikie commented on issue #17375:
URL: https://github.com/apache/airflow/issues/17375#issuecomment-945179813


   This issues is similar to #18304. I was able to workaround this as per https://github.com/apache/airflow/issues/18304#issuecomment-945172125


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org