[GitHub] [hudi] YuweiXiao commented on a diff in pull request #6680: [HUDI-4812] Lazy fetching partition path & file slice for HoodieFileIndex

[GitBox <gi...@apache.org>]

YuweiXiao commented on code in PR #6680:
URL: https://github.com/apache/hudi/pull/6680#discussion_r1022214084


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/SparkHoodieTableFileIndex.scala:
##########
@@ -237,70 +246,64 @@ class SparkHoodieTableFileIndex(spark: SparkSession,
     }
   }
 
-  private def listMatchingPartitionPathsInternal(partitionColumnNames: Seq[String],
-                                                 partitionColumnPredicates: Seq[Expression]): Seq[PartitionPath] = {
-    // NOTE: Here we try to to achieve efficiency in avoiding necessity to recursively list deep folder structures of
-    //       partitioned tables w/ multiple partition columns, by carefully analyzing provided partition predicates:
-    //
-    //       In cases when partition-predicates have
-    //         - The form of equality predicates w/ static literals (for ex, like `date = '2022-01-01'`)
-    //         - Fully specified proper prefix of the partition schema (ie fully binding first N columns
-    //           of the partition schema adhering to hereby described rules)
-    //
-    // We will try to exploit this specific structure, and try to reduce the scope of a
-    // necessary file-listings of partitions of the table to just the sub-folder under relative prefix
-    // of the partition-path derived from the partition-column predicates. For ex, consider following
-    // scenario:
-    //
-    // Table's partition schema (in-order):
-    //
-    //    country_code: string (for ex, 'us')
-    //    date: string (for ex, '2022-01-01')
-    //
-    // Table's folder structure:
-    //    us/
-    //     |- 2022-01-01/
-    //     |- 2022-01-02/
-    //     ...
-    //
-    // In case we have incoming query specifies following predicates:
-    //
-    //    `... WHERE country_code = 'us' AND date = '2022-01-01'`
-    //
-    // We can deduce full partition-path w/o doing a single listing: `us/2022-01-01`
-    if (areAllPartitionPathsCached || !shouldUsePartitionPathPrefixAnalysis(configProperties)) {
-      logDebug("All partition paths have already been cached, use it directly")
+  // NOTE: Here we try to to achieve efficiency in avoiding necessity to recursively list deep folder structures of
+  //       partitioned tables w/ multiple partition columns, by carefully analyzing provided partition predicates:
+  //
+  //       In cases when partition-predicates have
+  //         - The form of equality predicates w/ static literals (for ex, like `date = '2022-01-01'`)
+  //         - Fully specified proper prefix of the partition schema (ie fully binding first N columns
+  //           of the partition schema adhering to hereby described rules)
+  //
+  // We will try to exploit this specific structure, and try to reduce the scope of a
+  // necessary file-listings of partitions of the table to just the sub-folder under relative prefix
+  // of the partition-path derived from the partition-column predicates. For ex, consider following
+  // scenario:
+  //
+  // Table's partition schema (in-order):
+  //
+  //    country_code: string (for ex, 'us')
+  //    date: string (for ex, '2022-01-01')
+  //
+  // Table's folder structure:
+  //    us/
+  //     |- 2022-01-01/
+  //     |- 2022-01-02/
+  //     ...
+  //
+  // In case we have incoming query specifies following predicates:
+  //
+  //    `... WHERE country_code = 'us' AND date = '2022-01-01'`
+  //
+  // We can deduce full partition-path w/o doing a single listing: `us/2022-01-01`

Review Comment:
   looks like a typo. _w/o_ means _without_? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org