[GitHub] [arrow] bkietz commented on a change in pull request #8187: ARROW-10003: [C++] Create parent dir for any destination fs in CopyFiles

[GitBox <gi...@apache.org>]

bkietz commented on a change in pull request #8187:
URL: https://github.com/apache/arrow/pull/8187#discussion_r488129044



##########
File path: cpp/src/arrow/filesystem/filesystem.cc
##########
@@ -448,6 +449,13 @@ Status CopyFiles(const std::vector<FileLocator>& sources,
                            destinations.size(), " paths.");
   }
 
+  RETURN_NOT_OK(::arrow::internal::OptionalParallelFor(
+      use_threads, static_cast<int>(sources.size()), [&](int i) {
+        auto dest_dir = internal::GetAbstractPathParent(destinations[i].path).first;
+        return dest_dir.empty() ? Status::OK()
+                                : destinations[i].filesystem->CreateDir(dest_dir);

Review comment:
       This may be very redundant, for example when copying lots of files into a single destination directory. If we had FileLocator::ToUri or similar we could maintain an unordered_set to ensure each directory is created only once




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org