[GitHub] [airflow] tooptoop4 opened a new issue #8789: Ability to throttle scheduler bursts on single pool

[GitBox <gi...@apache.org>]

tooptoop4 opened a new issue #8789:
URL: https://github.com/apache/airflow/issues/8789


   Current issue is I have pools configured on same type of task where I can have up to max 100 pool slots of that task type running in parallel BUT I want ability to say don't START running more than 5 at a time. So ideally these tasks should be scheduled in steps.
   ie 
   at each scheduler iteration 1 (ie 9:03:30 am) 5 tasks scheduled for the pool
   at each scheduler iteration 2 (ie 9:04:00 am) 5 more tasks scheduled for the pool (so 10 currently running in the pool)
   at each scheduler iteration 3 (ie 9:04:30 am) 5 more tasks scheduled for the pool (so 15 currently running in the pool)
   at 9:04:42 am 3 tasks completed for the pool
   at each scheduler iteration 4 (ie 9:05:00 am) 5 more tasks scheduled for the pool (so 17 currently running in the pool)
   ..etc


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org