[GitHub] [gobblin] Will-Lo commented on a diff in pull request #3537: Add error handling for timeaware finder to handle scenarios where fil…

[GitBox <gi...@apache.org>]

Will-Lo commented on code in PR #3537:
URL: https://github.com/apache/gobblin/pull/3537#discussion_r949567548


##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/TimeAwareRecursiveCopyableDataset.java:
##########
@@ -138,8 +138,13 @@ protected List<FileStatus> getFilesAtPath(FileSystem fs, Path path, PathFilter f
   private List<FileStatus> recursivelyGetFilesAtDatePath(FileSystem fs, Path path, String traversedDatePath, PathFilter fileFilter,
       int level,  LocalDateTime startDate, LocalDateTime endDate, DateTimeFormatter formatter) throws IOException {
     List<FileStatus> fileStatuses = Lists.newArrayList();
-    Iterator<FileStatus> folderIterator = Arrays.asList(fs.listStatus(path)).iterator();
-
+    Iterator<FileStatus> folderIterator;
+    try {
+      folderIterator = Arrays.asList(fs.listStatus(path)).iterator();

Review Comment:
   By check before list status do you mean to check that all the folders match the date time format at that current level of granularity? Just following the previous behavior of this finder, which was to report an exception and return empty list instead of failing the job. Current issues are caused by failling the job and not ignoring badly formatted files.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org