[GitHub] [airflow] jedcunningham commented on a change in pull request #16075: Parse recently modified files even if just parsed

[GitBox <gi...@apache.org>]

jedcunningham commented on a change in pull request #16075:
URL: https://github.com/apache/airflow/pull/16075#discussion_r639251900



##########
File path: airflow/utils/dag_processing.py
##########
@@ -1057,11 +1057,15 @@ def prepare_file_path_queue(self):
             else:
                 file_paths.append(file_path)
 
-            # Find file paths that were recently processed
+            # Find file paths that were recently processed to exclude them
+            # from being added to file_path_queue
+            # unless they were modified recently and parsing mode is "modified_time"
             last_finish_time = self.get_last_finish_time(file_path)
+            file_modified_time = timezone.make_aware(datetime.fromtimestamp(files_with_mtime[file_path]))

Review comment:
       Won't this raise a `KeyError` if we aren't in mtime mode?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org