You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/06/10 11:14:12 UTC
[GitHub] [airflow] ashb opened a new issue #16366: Create DagRuns in queued state, and have scheduler set them to running
ashb opened a new issue #16366:
URL: https://github.com/apache/airflow/issues/16366
There have been various problems with the behaviour of max_active_runs not being respected (https://github.com/apache/airflow/issues/9975, https://github.com/apache/airflow/issues/14744, and we should try and fix this once and for all by changing how dag runs are created and run.
Currently creating a DagRun in the webserver puts it immediately in to "running" state, which can end up with too many DagRuns being executed, causing problems for the scheduler.
Clearing a DagRin can have similar problems.
We should change the behaviour as follows:
- If the webserver creates/clears and DagRun and would previously put it in to running state, it should put it in to "queued" instead
- Make SchedulerJob._create_dagruns_for_dags _also_ create DRs in queued state.
- Add a new stage to SchedulerJob._do_scheduling that checks For any queued DagRun if there is "space" to promote it to running
- We can remove any checks about currently_active_runs inside _schedule_dag_run function
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16366: Create DagRuns in queued state, and have scheduler set them to running
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16366:
URL: https://github.com/apache/airflow/issues/16366#issuecomment-975567233
> Solved by #16401 (according to PR description). Not sure why Github didn't auto close it :/
When there is more than one issue they should be mentioned with full syntax:
Closes :#XXX, Closes :#XXY
See "mutliple issues" in
https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #16366: Create DagRuns in queued state, and have scheduler set them to running
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #16366:
URL: https://github.com/apache/airflow/issues/16366#issuecomment-975578708
> CC: @ephraimbuddy ^^
Thanks!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal closed issue #16366: Create DagRuns in queued state, and have scheduler set them to running
Posted by GitBox <gi...@apache.org>.
eladkal closed issue #16366:
URL: https://github.com/apache/airflow/issues/16366
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #16366: Create DagRuns in queued state, and have scheduler set them to running
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #16366:
URL: https://github.com/apache/airflow/issues/16366#issuecomment-975489561
Solved by https://github.com/apache/airflow/pull/16401 (according to PR description). Not sure why Github didn't auto close it :/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16366: Create DagRuns in queued state, and have scheduler set them to running
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16366:
URL: https://github.com/apache/airflow/issues/16366#issuecomment-975567499
CC: @ephraimbuddy ^^
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org