[GitHub] [crunch] jonhemphill commented on a change in pull request #23: CRUNCH-683 Avoid unnecessary listStatus calls from getPathSize computation

[GitBox <gi...@apache.org>]

jonhemphill commented on a change in pull request #23: CRUNCH-683 Avoid unnecessary listStatus calls from getPathSize computation
URL: https://github.com/apache/crunch/pull/23#discussion_r278546913
 
 

 ##########
 File path: crunch-core/src/main/java/org/apache/crunch/io/SourceTargetHelper.java
 ##########
 @@ -41,17 +41,23 @@ public static long getPathSize(FileSystem fs, Path path) throws IOException {
     }
     long size = 0;
     for (FileStatus status : stati) {
-      if (status.isDir()) {
-        for (FileStatus st : fs.listStatus(status.getPath())) {
-          size += getPathSize(fs, st.getPath());
-        }
-      } else {
-        size += status.getLen();
-      }
+      size += getPathSize(fs, status);
     }
     return size;
   }
-  
+
+  private static long getPathSize(final FileSystem fs, final FileStatus status) throws IOException {
 
 Review comment:
   Thanks for the feedback @steveloughran. I have it updated locally to use `listFiles` as you suggest and it is working well, I just need to update the tests now before commiting.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services