You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/10/11 02:39:27 UTC
[GitHub] [hudi] garyli1019 commented on a change in pull request #3768: [HUDI-2494] Fixing glob pattern to skip all hoodie meta paths
garyli1019 commented on a change in pull request #3768:
URL: https://github.com/apache/hudi/pull/3768#discussion_r725753489
##########
File path: hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieSparkUtils.scala
##########
@@ -64,8 +64,24 @@ object HoodieSparkUtils extends SparkAdapterSupport {
* [[org.apache.spark.deploy.SparkHadoopUtil]] becomes private since Spark 3.0.0 and hence we had to copy it locally.
*/
def globPath(fs: FileSystem, pattern: Path): Seq[Path] = {
- Option(fs.globStatus(pattern)).map { statuses =>
- statuses.map(_.getPath.makeQualified(fs.getUri, fs.getWorkingDirectory)).toSeq
+ // find base path to assist in skipping meta paths
Review comment:
This method seems copied from Spark. IMO we could change the name to `globPathWithoutMeta` or have a separate method to `filterMeta()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org