[GitHub] yrqls21 edited a comment on issue #2986: [AIRFLOW-2027] Only trigger sleep in scheduler after all files have parsed

[GitBox <gi...@apache.org>]

yrqls21 edited a comment on issue #2986: [AIRFLOW-2027] Only trigger sleep in scheduler after all files have parsed
URL: https://github.com/apache/incubator-airflow/pull/2986#issuecomment-412637527
 
 
   @aoen 
   Agree on sleep for a minimal time interval(max(0, 1s-duration)), it should not be a huge problem to prolong the scheduler loop time to at least one second every loop( have that in my branch for a while and looks good until now). 
   
   I'm not so sure about monotonically increasing, what I can see is that if we have parse files faster than we set file path, then if file was removed/have import error and not updating the last_finish_time in the 2nd parsing round, we'll still use the last_finish_time from the 1st round and get some non-desirable duration value. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services