[GitHub] [arrow] westonpace commented on a diff in pull request #12625: ARROW-15587: [C++] Add support for all options specified by substrait::ReadRel::LocalFiles::FileOrFiles

[GitBox <gi...@apache.org>]

westonpace commented on code in PR #12625:
URL: https://github.com/apache/arrow/pull/12625#discussion_r841969110


##########
cpp/src/arrow/dataset/discovery.cc:
##########
@@ -134,8 +135,26 @@ Result<std::shared_ptr<DatasetFactory>> FileSystemDatasetFactory::Make(
 Result<std::shared_ptr<DatasetFactory>> FileSystemDatasetFactory::Make(
     std::shared_ptr<fs::FileSystem> filesystem, const std::vector<fs::FileInfo>& files,
     std::shared_ptr<FileFormat> format, FileSystemFactoryOptions options) {
+  // Discover files in directories and globs
+  std::vector<fs::FileInfo> discovered_files;
+  for (const auto& file : files) {
+    if (file.IsDirectory()) {
+      fs::FileSelector file_selector;
+      file_selector.base_dir = file.dir_name();
+      file_selector.recursive = true;
+      ARROW_ASSIGN_OR_RAISE(auto folder_files, filesystem->GetFileInfo(file_selector));
+      std::move(folder_files.begin(), folder_files.end(),
+                std::back_inserter(discovered_files));
+    } else if (file.IsGlob()) {
+      ARROW_ASSIGN_OR_RAISE(auto files, filesystem->GetFileInfoGlob(file));
+      std::move(files.begin(), files.end(), std::back_inserter(discovered_files));
+    } else if (file.IsFile()) {
+      discovered_files.emplace_back(file);
+    }
+  }
+

Review Comment:
   Ok, yes, I think an enum approach is fine.  I would maybe call it `SelectorType` instead of `FileType` since we already have something named `FileType`.  I would have a slight preference to not reuse `FileType` because that will lead to some confusion (users creating a selector will wonder what "unknown" means and users getting fileinfo will wonder what "glob" means).
   
   Also, we will want to make sure it is clear (in comments at least to start but later in documentation) the meaning of the fields.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org