[GitHub] [airflow] vapiravfif edited a comment on issue #19622: Tasks get stuck in "scheduled" state and starved when dags with huge amount of tasks is scheduled

[GitBox <gi...@apache.org>]

vapiravfif edited a comment on issue #19622:
URL: https://github.com/apache/airflow/issues/19622#issuecomment-976280104


   @halilduygulu Thank you for verifying this on newer version, haven't gotten to this yet. Ultimately, increasing only the `parallelism` value should solve this (in my case I bumped it to 1024, just to be sure). Another parameter affectin this is `max_tis_per_query`, which is 512 by default. Scheduler takes minimum value between them.
   
   The origin of the issue seems to be in scheduler limiting the query of tasks in "scheduled" state and waiting to be queued by the amount of potentially available slots. But then some of the tasks can be still limited by dag concurrency and other constraints, resulting that tasks which are not limited not being picked up for a long time...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org