[GitHub] [airflow] victor-mariano-leite commented on issue #13542: Task stuck in "scheduled" or "queued" state, pool has all slots queued, nothing is executing

[GitBox <gi...@apache.org>]

victor-mariano-leite commented on issue #13542:
URL: https://github.com/apache/airflow/issues/13542#issuecomment-1098582282

   For those having problems with MWAA, I had this error today, and couldn't wait for 2.2.5 release in MWAA to finish my company migration project, so since we have 17 DAGs in my company, with 8-9 steps as median of tasks (one have 100+ tasks, for each table in our DB, runs a import/CDC and validation task), being all those 17 running once a day at night. 
   
   I went a bit extreme with reducing the load on the scheduler, and looks like it's working properly after a few tests today.
   
   If anyone want to experiment and are having the same problem with similar settings, here's the configurations i've changed, using `mw1.medium`:
   
   | Configuration                              | Value |
   |--------------------------------------------|-------|
   | celery.sync_parallelism                    | 1     |
   | celery.worker_autoscale                    | 5,5   |
   | core.min_serialized_dag_fetch_interval     | 20    |
   | core.min_serialized_dag_update_interval    | 60    |
   | scheduler.catchup_by_default               | False |
   | scheduler.job_heartbeat_sec                | 10    |
   | scheduler.max_dagruns_per_loop_to_schedule | 10    |
   | scheduler.max_dagruns_to_create_per_loop   | 5     |
   | scheduler.max_tis_per_query                | 128   |
   | scheduler.min_file_process_interval        | 60    |
   | scheduler.parsing_processes                | 1     |
   | scheduler.scheduler_idle_sleep_time        | 5     |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org