[GitHub] [airflow] pavansharma36 commented on pull request #27060: 27010 : Add new files to parsing queue on every loop of dag processsing

[GitBox <gi...@apache.org>]

pavansharma36 commented on PR #27060:
URL: https://github.com/apache/airflow/pull/27060#issuecomment-1309036169

   @ashb 
   
   Existing implementation adds new file to parsing queue only if parsing queue is empty.
   Consider we set refresh interval to 5 seconds and have 10k existing dag files which takes around 5 minutes to parse, Even though file paths are refreshed every 5 seconds new files are not added to parsing queue until all 10k files are parsed and queue becomes empty
   
   This change to make sure if file_path is refreshed on list_interval config, new files are added to parsing queue without waiting for existing file parsing.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org