[GitHub] [arrow] lidavidm commented on a diff in pull request #14747: ARROW-18113: [C++] Use RandomAccessFile::ReadManyAsync for cloud file systems

[GitBox <gi...@apache.org>]

lidavidm commented on code in PR #14747:
URL: https://github.com/apache/arrow/pull/14747#discussion_r1034053316


##########
cpp/src/arrow/filesystem/s3fs.h:
##########
@@ -167,6 +168,12 @@ struct ARROW_EXPORT S3Options {
   /// delay between retries.
   std::shared_ptr<S3RetryStrategy> retry_strategy;
 
+  /// Coalesce options for requesting multiple reads at once
+  io::CoalesceOptions coalesceOptions = io::CoalesceOptions::Defaults();

Review Comment:
   nit: follow the naming conventions (snake_case)



##########
cpp/src/arrow/filesystem/s3fs.cc:
##########
@@ -1128,6 +1129,14 @@ class ObjectInputFile final : public io::RandomAccessFile {
     return bytes_read;
   }
 
+  std::vector<Future<std::shared_ptr<Buffer>>> ReadManyAsync(
+      const io::IOContext& ctx, const std::vector<io::ReadRange>& ranges) override {
+    return RandomAccessFile::ReadManyAsync(
+        ctx, io::internal::CoalesceReadRanges(std::move(ranges),
+                                              coalesceOptions.hole_size_limit,
+                                              coalesceOptions.range_size_limit));

Review Comment:
   Since coalescing may merge ranges, you'll need to post-process things so that the returned futures map 1:1 with the original ranges. (May need to refactor things so you don't have to do the coalescing step twice?)



##########
cpp/src/arrow/filesystem/s3fs.h:
##########
@@ -167,6 +168,12 @@ struct ARROW_EXPORT S3Options {
   /// delay between retries.
   std::shared_ptr<S3RetryStrategy> retry_strategy;
 
+  /// Coalesce options for requesting multiple reads at once
+  io::CoalesceOptions coalesceOptions = io::CoalesceOptions::Defaults();
+
+  int64_t hole_size_limit = 1;
+  int64_t range_size_limit = 100;

Review Comment:
   Why do we need these separately?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org