[GitHub] [airflow] jedcunningham edited a comment on pull request #19747: Fix Tasks getting stuck in scheduled state

[GitBox <gi...@apache.org>]

jedcunningham edited a comment on pull request #19747:
URL: https://github.com/apache/airflow/pull/19747#issuecomment-1028455240


   Okay, I benchmarked 2.2.3, #19747 on latest main, and #20391 on latest main with:
   
   - 100 DAGs, 10 runs each, 10 task per DAG for 10k total TIs
   - 2 workers (1 CPU core each)
   - 1 scheduler (1 CPU core)
   
   It took each this long to get through them all:
   
   * 2.2.3: 51 minutes
   * 19747: 38 minutes
   * 20391: 38 minutes
   
   (I should say, yes, they took the same amount of time, I double checked my data)
   
   #19747 however does spew this into the scheduler logs ~16x per second (quite chatty!):
   
   ```
   [2022-02-02 23:21:32,947] {scheduler_job.py:522} INFO - Setting the following tasks to queued state:
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org