[GitHub] [airflow] tienhung2812 commented on issue #13542: Task stuck in "scheduled" or "queued" state, pool has all slots queued, nothing is executing

[GitBox <gi...@apache.org>]

tienhung2812 commented on issue #13542:
URL: https://github.com/apache/airflow/issues/13542#issuecomment-921858288


   Hi, just got this issue again on `2.1.3` but the behavior quite different, all of schedulers stuck into a DAG check loop, but restart the scheduler will fix it, all other DAG as stuck in `scheduled`
   
   ```
    [scheduled]> since the number of tasks running or queued from DAG abc_dag is >= to the DAG's task concurrency limit of 16
   airflow_scheduler.2.bj0t349vgttt@ip-172-30-2-178    | [2021-09-17 14:29:07,511] {scheduler_job.py:410} INFO - DAG abc_dag has 16/16 running and queued tasks
   airflow_scheduler.2.bj0t349vgttt@ip-172-30-2-178    | [2021-09-17 14:29:07,512] {scheduler_job.py:417} INFO - Not executing <TaskInstance: abc_dag.aa.aa_20.collector_v2 2021-09-17 11:30:00+00:00 [scheduled]> since the number of tasks running or queued from DAG abc_dag is >= to the DAG's task concurrency limit of 16
   airflow_scheduler.2.bj0t349vgttt@ip-172-30-2-178    | [2021-09-17 14:29:07,512] {scheduler_job.py:410} INFO - DAG abc_dag has 16/16 running and queued tasks
   airflow_scheduler.2.bj0t349vgttt@ip-172-30-2-178    | [2021-09-17 14:29:07,513] {scheduler_job.py:417} INFO - Not executing <TaskInstance: abc_dag.aa.ff_46.collector_v2 2021-09-17 11:30:00+00:00 [scheduled]> since the number of tasks running or queued from DAG abc_dag is >= to the DAG's task concurrency limit of 16
   airflow_scheduler.2.bj0t349vgttt@ip-172-30-2-178    | [2021-09-17 14:29:07,513] {scheduler_job.py:410} INFO - DAG abc_dag has 16/16 running and queued tasks`
   airflow_scheduler.2.bj0t349vgttt@ip-172-30-2-178    | [2021-09-17 14:29:07,514] {scheduler_job.py:417} INFO - Not executing <TaskInstance: abc_dag.ff.gg_121.collector_v2 2021-09-17 11:30:00+00:00 [scheduled]> since the number of tasks running or queued from DAG abc_dag is >= to the DAG's task concurrency limit of 16
   ```
   
   The current system have about >= 150 active dags but we have a`abc_dag` have more than 500 tasks inside
   I have already increase the scheduler config to handling more task
   - max_tis_per_query: 512
   - max_dagruns_to_create_per_loop: 50
   - max_dagruns_per_loop_to_schedule: 70
   I have also add a scheduler healthcheck but it can not detect this issue 
   
   Environment:
   - Celery Executor
   - MySQL 8.0.23-14
   - Airflow: 2.1.3 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org