[GitHub] [airflow] KevinYang21 commented on issue #1906: [AIRFLOW-536] Schedule all pending DAG runs in a single scheduler loop

[GitBox <gi...@apache.org>]

KevinYang21 commented on issue #1906: [AIRFLOW-536] Schedule all pending DAG runs in a single scheduler loop
URL: https://github.com/apache/airflow/pull/1906#issuecomment-486513814
 
 
   @ashb I do think it worth resurrect this PR, but the change needed would be more complicated than what was proposed in this PR.
   
   I don't know how this change is related to the workers got saturated issue from @bolkedebruin, maybe the code base was very different back then. I'm curious to know what was the concerns back then but from my point of view we should be good, only that we need to be careful about the way we create new DAG run--the current implementation would be problematic if we have DAG with large number of dag run pending as we do the `max_active_runs` later on and creating all the pending DAG runs one by one is not the most efficient way.
   
   About the concern of task scheduling, I'm 100% following what @vijaysbhat was describing but this change would definitely be very helpful for Airflow to schedule task faster. Esp. for those DAGs want to catch up from an old execution_date and those cluster with long scheduler DAG file parsing interval.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services