[GitHub] [arrow] ArianaVillegas commented on a diff in pull request #12625: ARROW-15587: [C++] Add support for all options specified by substrait::ReadRel::LocalFiles::FileOrFiles

[GitBox <gi...@apache.org>]

ArianaVillegas commented on code in PR #12625:
URL: https://github.com/apache/arrow/pull/12625#discussion_r842046641


##########
cpp/src/arrow/dataset/discovery.cc:
##########
@@ -134,8 +135,26 @@ Result<std::shared_ptr<DatasetFactory>> FileSystemDatasetFactory::Make(
 Result<std::shared_ptr<DatasetFactory>> FileSystemDatasetFactory::Make(
     std::shared_ptr<fs::FileSystem> filesystem, const std::vector<fs::FileInfo>& files,
     std::shared_ptr<FileFormat> format, FileSystemFactoryOptions options) {
+  // Discover files in directories and globs
+  std::vector<fs::FileInfo> discovered_files;
+  for (const auto& file : files) {
+    if (file.IsDirectory()) {
+      fs::FileSelector file_selector;
+      file_selector.base_dir = file.dir_name();
+      file_selector.recursive = true;
+      ARROW_ASSIGN_OR_RAISE(auto folder_files, filesystem->GetFileInfo(file_selector));
+      std::move(folder_files.begin(), folder_files.end(),
+                std::back_inserter(discovered_files));
+    } else if (file.IsGlob()) {
+      ARROW_ASSIGN_OR_RAISE(auto files, filesystem->GetFileInfoGlob(file));
+      std::move(files.begin(), files.end(), std::back_inserter(discovered_files));
+    } else if (file.IsFile()) {
+      discovered_files.emplace_back(file);
+    }
+  }
+

Review Comment:
   Ok, I understand. So, should I go ahead and perform the changes? Or should we be waiting for someone else opinion?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org