[jira] [Created] (ARROW-8884) [C++] Listing files with S3FileSystem is slow

["Francois Saint-Jacques (Jira)" <ji...@apache.org>]

Francois Saint-Jacques created ARROW-8884:
---------------------------------------------

             Summary: [C++] Listing files with S3FileSystem is slow
                 Key: ARROW-8884
                 URL: https://issues.apache.org/jira/browse/ARROW-8884
             Project: Apache Arrow
          Issue Type: Improvement
          Components: C++
            Reporter: Francois Saint-Jacques


Listing files on S3 is slow due to the recursive nature of the algorithm.

The following change modifies the behavior of the S3Result to include all objects but no "grouping" (directories). This lower dramatically the number of HTTP calls. 
{code:c++}
diff --git a/cpp/src/arrow/filesystem/s3fs.cc b/cpp/src/arrow/filesystem/s3fs.cc
index 70c87f46ec..98a40b17a2 100644
--- a/cpp/src/arrow/filesystem/s3fs.cc
+++ b/cpp/src/arrow/filesystem/s3fs.cc
@@ -986,7 +986,7 @@ class S3FileSystem::Impl {
     if (!prefix.empty()) {
       req.SetPrefix(ToAwsString(prefix) + kSep);
     }
-    req.SetDelimiter(Aws::String() + kSep);
+    // req.SetDelimiter(Aws::String() + kSep);
     req.SetMaxKeys(kListObjectsMaxKeys);
 
     while (true) {

{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)