You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2018/08/13 13:50:43 UTC
[GitHub] aoen commented on issue #2986: [AIRFLOW-2027] Only trigger sleep in
scheduler after all files have parsed
aoen commented on issue #2986: [AIRFLOW-2027] Only trigger sleep in scheduler after all files have parsed
URL: https://github.com/apache/incubator-airflow/pull/2986#issuecomment-412524787
There seems to be a couple of problems:
1. If the number of files (can be fixed by sleeping for min_sleep even if there are no files)
2. I have heard reports that some files can return a parsing time that is monotonically increasing for some reason (e.g. file actually parses in 1s each loop, but the reported duration seems to use the very time the file was parsed as the start time instead of the last time), I haven't confirmed this but it sounds problematic.
To unblock the release I'm reverting this PR for now. It should be re-added with tests/mocking.
@bolkedebruin @yrqls21
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services