You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/13 12:44:02 UTC
[GitHub] [airflow] ashb opened a new issue #11498: Performan "mini scheduling run" after task has finished
ashb opened a new issue #11498:
URL: https://github.com/apache/airflow/issues/11498
In order to further reduce intra-dag task scheduling lag we can add a further optimization: when a task has just finshed executing (success or failure) we can look at the downstream tasks of the task that just finished, and then make scheduling decisions for those tasks there -- we've already got the dag loaded, and we know they are _likely_ actionable as we just finished. We should set tasks to scheduled if we can (but no further, i.e. not to queued).
Upstream_failed may also not be safe as that would run callbacks. Think about this.
We'll need a config option to disable this, as it _could_ lead to task starvation of other dags
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #11498: Perform "mini scheduling run" after task has finished
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #11498:
URL: https://github.com/apache/airflow/issues/11498#issuecomment-707717945
Good idea!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb closed issue #11498: Perform "mini scheduling run" after task has finished
Posted by GitBox <gi...@apache.org>.
ashb closed issue #11498:
URL: https://github.com/apache/airflow/issues/11498
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #11498: Perform "mini scheduling run" after task has finished
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #11498:
URL: https://github.com/apache/airflow/issues/11498#issuecomment-707724677
We've already tested a hacky version of this, and it' makes a difference. (This was one of the two remaining items from https://github.com/apache/airflow/issues/9630)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org