[GitHub] [pinot] walterddr commented on a diff in pull request #9466: collect file info like mtime, length while listing files for free

[GitBox <gi...@apache.org>]

walterddr commented on code in PR #9466:
URL: https://github.com/apache/pinot/pull/9466#discussion_r1021791705


##########
pinot-plugins/pinot-file-system/pinot-s3/src/main/java/org/apache/pinot/plugin/filesystem/S3PinotFS.java:
##########
@@ -435,8 +437,48 @@ public long length(URI fileUri)
   @Override
   public String[] listFiles(URI fileUri, boolean recursive)
       throws IOException {
+    ImmutableList.Builder<String> builder = ImmutableList.builder();
+    visitFiles(fileUri, recursive, s3Object -> {
+      // TODO: Looks like S3PinotFS filters out directories, inconsistent with the other implementations.
+      // Only add files and not directories

Review Comment:
   seems like this would break if we enable recursive file ingestion on the ingestion task config?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org