You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "smdsbz (via GitHub)" <gi...@apache.org> on 2025/01/20 07:14:21 UTC

[PR] Iterative list for OSS [paimon]

smdsbz opened a new pull request, #4952:
URL: https://github.com/apache/paimon/pull/4952

   <!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
   
   ### Purpose
   
   <!-- Linking this pull request to the issue -->
   Linked issue: close #4951 
   
   <!-- What is the purpose of the change -->
   Take advantage of OSS's native batched list API to provide an optimized implementation of listFilesIterative for OSS.
   
   ### Tests
   
   <!-- List UT and IT cases to verify this change -->
   None.
   
   (I can't seem to find any existing tests for various `FileIO`s.)
   
   ### API and Format
   
   <!-- Does this change affect API or storage format -->
   N/A
   
   ### Documentation
   
   <!-- Does this change introduce a new feature -->
   N/A


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Optimized iterative list implementations [paimon]

Posted by "smdsbz (via GitHub)" <gi...@apache.org>.
smdsbz commented on PR #4952:
URL: https://github.com/apache/paimon/pull/4952#issuecomment-2602171712

   > Can you add all implementations for hadoop, s3 and oss?
   
   No problem.
   
   > Then, you can add tests.
   
   Where may I find the OSS, S3 buckets for testing? Or I'll have to mock their implementation?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Optimized iterative list implementations [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #4952:
URL: https://github.com/apache/paimon/pull/4952


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Iterative list for OSS [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #4952:
URL: https://github.com/apache/paimon/pull/4952#issuecomment-2601902676

   Can you add all implementations for hadoop, s3 and oss?
   
   Then, you can add tests.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Optimized iterative list implementations [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4952:
URL: https://github.com/apache/paimon/pull/4952#discussion_r1922983988


##########
paimon-common/src/main/java/org/apache/paimon/fs/hadoop/HadoopFileIO.java:
##########
@@ -104,6 +105,29 @@ public FileStatus[] listStatus(Path path) throws IOException {
         return statuses;
     }
 
+    @Override
+    public RemoteIterator<FileStatus> listFilesIterative(Path path, boolean recursive)

Review Comment:
   Add tests for hadoop file io?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Optimized iterative list implementations [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #4952:
URL: https://github.com/apache/paimon/pull/4952#issuecomment-2603641353

   +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org