You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "cshuo (via GitHub)" <gi...@apache.org> on 2024/10/21 02:41:07 UTC

[PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

cshuo opened a new pull request, #12132:
URL: https://github.com/apache/hudi/pull/12132

   ### Change Logs
   
   This PR introduces a new partition pruner for Flink source based on the Partition Stats Index.
   
   Before this PR, Flink source (batch or streaming) uses partition filters pushed down to build partition pruner and filter irrelevant partitions. Then, Column Stats Index is used to build data pruner to do the file-level data skipping. HUDI-7144 introduced the partition-level column stats, we can use the stats to prune partitions just like the way files are pruned.
   
   Main changes:
   * Add utilities to fetch Partition Stats Index data for Flink source.
   * Add a new partition pruner `ColumnStatsPartitionPruner`.
   * Add new config `read.partition.data.skipping.enabled` to enabled partition stats pruner, `false` by default.
   
   ### Impact
   Enhance the data skipping ability for Flink source by introducing a new partition pruner based on Partition Stats Index.
   
   ### Risk level (write none, low medium or high below)
   low
   
   ### Documentation Update
   none 
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #12132:
URL: https://github.com/apache/hudi/pull/12132#discussion_r1808648628


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -378,6 +378,14 @@ private FlinkOptions() {
       .withDescription("Enables data-skipping allowing queries to leverage indexes to reduce the search space by "
           + "skipping over files");
 
+  @AdvancedConfig
+  public static final ConfigOption<Boolean> READ_PARTITION_DATA_SKIPPING_ENABLED = ConfigOptions
+      .key("read.partition.data.skipping.enabled")

Review Comment:
   The main purpose here is to provide the ability to enable different indexes separately, just in case that there are unexpected behaviors in some indexes.  Maybe we can change the code to that all indexes are enabled by default when data skipping is enabled, and introduce a new config option 'read.data.skipping.excluded.indexes' to disable some indexes if necessary. WDYT?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12132:
URL: https://github.com/apache/hudi/pull/12132#discussion_r1808053349


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/stats/ColumnStatsIndices.java:
##########
@@ -85,16 +89,109 @@ public class ColumnStatsIndices {
   private ColumnStatsIndices() {
   }
 
-  public static List<RowData> readColumnStatsIndex(String basePath, HoodieMetadataConfig metadataConfig, String[] targetColumns) {
+  public static Set<String> candidatePartitionsInMetadataTable(
+      String basePath,
+      HoodieMetadataConfig metadataConfig,
+      RowType rowType,
+      @Nullable DataPruner dataPruner,
+      List<String> filesOrPartitions) {
+    if (dataPruner == null) {
+      return null;
+    }
+    final List<RowData> indexRows =
+        ColumnStatsIndices.readPartitionStatsIndex(basePath, metadataConfig, dataPruner.getReferencedCols());
+    return candidatesInMetadataTable(rowType, dataPruner, indexRows, filesOrPartitions);
+  }
+
+  public static Set<String> candidateFilesInMetadataTable(
+      String basePath,
+      HoodieMetadataConfig metadataConfig,
+      RowType rowType,
+      @Nullable DataPruner dataPruner,
+      List<String> filesOrPartitions) {
+    if (dataPruner == null) {
+      return null;
+    }
+    final List<RowData> indexRows =
+        ColumnStatsIndices.readFileColumnStatsIndex(basePath, metadataConfig, dataPruner.getReferencedCols());
+    return candidatesInMetadataTable(rowType, dataPruner, indexRows, filesOrPartitions);
+  }
+
+  /**
+   * Computes pruned list of candidate base-files' names based on provided list of data filters.
+   * conditions, by leveraging Metadata Table's Column Statistics index (hereon referred as ColStats for brevity)
+   * bearing "min", "max", "num_nulls" statistics for all columns.
+   *
+   * <p>NOTE: This method has to return complete set of candidate files, since only provided candidates will
+   * ultimately be scanned as part of query execution. Hence, this method has to maintain the
+   * invariant of conservatively including every base-file's name, that is NOT referenced in its index.
+   *
+   * <p>The {@code filters} must all be simple.
+   *
+   * @param dataPruner the data pruner built from push-down filters.

Review Comment:
   supplement comments for all the parameters.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12132:
URL: https://github.com/apache/hudi/pull/12132#discussion_r1814090420


##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/ITTestHoodieDataSource.java:
##########
@@ -507,6 +508,61 @@ void testStreamReadWithDeletes() throws Exception {
     assertRowsEquals(result, expected, true);
   }
 
+  @ParameterizedTest
+  @MethodSource("executionModeAndTableTypeParams")
+  void testReadWithPartitionStatsPrune(ExecMode execMode, HoodieTableType tableType) throws Exception {

Review Comment:
   instead of execution mode, can we test the combination of hiveStylePartitioning and table type.
   
   prune -> pruning



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12132:
URL: https://github.com/apache/hudi/pull/12132#discussion_r1814088198


##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/source/TestIncrementalInputSplits.java:
##########
@@ -348,6 +358,51 @@ void testInputSplitsWithPartitionPruner(
     assertEquals(expectedPartitions, partitions);
   }
 
+  @Test
+  void testInputSplitsWithPartitionStatsPruner() throws Exception {
+    Configuration conf = TestConfigurations.getDefaultConf(basePath);
+    conf.set(FlinkOptions.READ_AS_STREAMING, true);
+    conf.set(FlinkOptions.READ_DATA_SKIPPING_ENABLED, true);
+    conf.setBoolean(HoodieMetadataConfig.ENABLE_METADATA_INDEX_PARTITION_STATS.key(), true);
+    TestData.writeData(TestData.DATA_SET_INSERT, conf);
+
+    // uuid > 'id5' and age < 30, only column stats of 'par3' matches the filter.
+    ColumnStatsProbe columnStatsProbe =
+        ColumnStatsProbe.newInstance(Arrays.asList(
+            new CallExpression(
+                FunctionIdentifier.of("greaterThan"),
+                BuiltInFunctionDefinitions.GREATER_THAN,
+                Arrays.asList(
+                    new FieldReferenceExpression("uuid", DataTypes.STRING(), 0, 0),
+                    new ValueLiteralExpression("id5", DataTypes.STRING().notNull())
+                ),
+                DataTypes.BOOLEAN()),
+            new CallExpression(
+                FunctionIdentifier.of("lessThan"),
+                BuiltInFunctionDefinitions.LESS_THAN,
+                Arrays.asList(
+                    new FieldReferenceExpression("age", DataTypes.INT(), 2, 2),
+                    new ValueLiteralExpression(30, DataTypes.INT().notNull())
+                ),
+                DataTypes.BOOLEAN())));
+
+    PartitionPruners.PartitionPruner partitionPruner =
+        PartitionPruners.builder().rowType(TestConfigurations.ROW_TYPE).basePath(basePath).conf(conf).columnStatsProbe(columnStatsProbe).build();
+    IncrementalInputSplits iis = IncrementalInputSplits.builder()
+        .conf(conf)
+        .path(new Path(basePath))
+        .rowType(TestConfigurations.ROW_TYPE)
+        .partitionPruner(partitionPruner)
+        .build();
+    metaClient = HoodieTableMetaClient.builder()

Review Comment:
   can we use `HoodieTestUtils.init(basePath, HoodieTableType.COPY_ON_WRITE)` here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12132:
URL: https://github.com/apache/hudi/pull/12132#issuecomment-2436670664

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9bff28bbdd59593f5c50bdbfca7af57543abd4be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1296",
       "triggerID" : "9bff28bbdd59593f5c50bdbfca7af57543abd4be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5f7b6934482578550b592c30e015bdc7bafea52a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1350",
       "triggerID" : "5f7b6934482578550b592c30e015bdc7bafea52a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7b9a50506e0e16f49da45498b1dd79e99300ac3a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1353",
       "triggerID" : "7b9a50506e0e16f49da45498b1dd79e99300ac3a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "06a5ca03ecb59ae6fb9b33de62fdb1e30bd747c3",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1393",
       "triggerID" : "06a5ca03ecb59ae6fb9b33de62fdb1e30bd747c3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "efde4491a94d517528c04c94449be0aee37d262e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "efde4491a94d517528c04c94449be0aee37d262e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 06a5ca03ecb59ae6fb9b33de62fdb1e30bd747c3 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1393) 
   * efde4491a94d517528c04c94449be0aee37d262e UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on PR #12132:
URL: https://github.com/apache/hudi/pull/12132#issuecomment-2430593717

   @danny0405 please take another look, thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12132:
URL: https://github.com/apache/hudi/pull/12132#discussion_r1809723198


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -378,6 +378,14 @@ private FlinkOptions() {
       .withDescription("Enables data-skipping allowing queries to leverage indexes to reduce the search space by "
           + "skipping over files");
 
+  @AdvancedConfig
+  public static final ConfigOption<Boolean> READ_PARTITION_DATA_SKIPPING_ENABLED = ConfigOptions
+      .key("read.partition.data.skipping.enabled")

Review Comment:
   We already have `read.data.skipping.enabled`, I don't think there is necessity to add more specific options.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12132:
URL: https://github.com/apache/hudi/pull/12132#discussion_r1809732826


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/stats/ColumnStatsIndices.java:
##########
@@ -85,16 +89,109 @@ public class ColumnStatsIndices {
   private ColumnStatsIndices() {
   }
 
-  public static List<RowData> readColumnStatsIndex(String basePath, HoodieMetadataConfig metadataConfig, String[] targetColumns) {
+  public static Set<String> candidatePartitionsInMetadataTable(

Review Comment:
   Can we have some basic abstractions so that it is more easy for extention, for e.g. the RLI support in the near future.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12132:
URL: https://github.com/apache/hudi/pull/12132#discussion_r1811625183


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/FileIndex.java:
##########
@@ -83,8 +77,8 @@ private FileIndex(StoragePath path, Configuration conf, RowType rowType, DataPru
     this.rowType = rowType;
     this.hadoopConf = HadoopConfigurations.getHadoopConf(conf);
     this.tableExists = StreamerUtil.tableExists(path.toString(), hadoopConf);
-    this.metadataConfig = metadataConfig(conf);
-    this.dataPruner = isDataSkippingFeasible(conf.getBoolean(FlinkOptions.READ_DATA_SKIPPING_ENABLED)) ? dataPruner : null;
+    this.metadataConfig = StreamerUtil.metadataConfig(conf);
+    this.dataPruner = isDataSkippingFeasible(conf.get(FlinkOptions.READ_DATA_SKIPPING_ENABLED)) ? dataPruner : null;

Review Comment:
   Let's rename the `DatatPruner` to `ColStatsProbe`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12132:
URL: https://github.com/apache/hudi/pull/12132#discussion_r1814088472


##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/source/TestIncrementalInputSplits.java:
##########
@@ -348,6 +358,51 @@ void testInputSplitsWithPartitionPruner(
     assertEquals(expectedPartitions, partitions);
   }
 
+  @Test
+  void testInputSplitsWithPartitionStatsPruner() throws Exception {

Review Comment:
   can we test both cow and mor table



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12132:
URL: https://github.com/apache/hudi/pull/12132#discussion_r1808051472


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/prune/PartitionPruners.java:
##########
@@ -125,6 +138,154 @@ public Set<String> filter(Collection<String> partitions) {
     }
   }
 
+  /**
+   * ColumnStats partition pruner for hoodie table source which enables partition stats index.
+   *
+   * <p>Note: The data of new partitions created after the job starts could be read if they match the
+   * filter conditions.
+   */
+  public static class ColumnStatsPartitionPruner implements PartitionPruner {
+    private final String basePath;
+    private final HoodieMetadataConfig metadataConfig;
+    private final DataPruner dataPruner;
+    private final RowType rowType;
+
+    public ColumnStatsPartitionPruner(
+        RowType rowType,
+        String basePath,
+        HoodieMetadataConfig metadataConfig,
+        DataPruner dataPruner) {
+      this.rowType = rowType;
+      this.basePath = basePath;
+      this.metadataConfig = metadataConfig;
+      this.dataPruner = dataPruner;
+    }
+
+    @Override
+    public Set<String> filter(Collection<String> partitions) {
+      Set<String> candidatePartitions = ColumnStatsIndices.candidatePartitionsInMetadataTable(
+          basePath, metadataConfig, rowType, dataPruner, new ArrayList<>(partitions));
+      if (candidatePartitions == null) {
+        return new HashSet<>(partitions);
+      }
+      return partitions.stream().filter(candidatePartitions::contains).collect(Collectors.toSet());
+    }
+  }
+
+  /**
+   * Chained partition pruner for hoodie table source which combines multiple partition pruners.
+   */
+  public static class ChainedPartitionPruner implements PartitionPruner {
+    private final List<PartitionPruner> pruners;
+
+    public ChainedPartitionPruner(List<PartitionPruner> pruners) {
+      this.pruners = pruners;
+    }
+
+    @Override
+    public Set<String> filter(Collection<String> partitions) {
+      for (PartitionPruner pruner: pruners) {
+        partitions = pruner.filter(partitions);
+      }
+      return new HashSet<>(partitions);
+    }
+  }
+
+  public static Builder builder() {
+    return new Builder();
+  }
+
+  public static class Builder {
+    private RowType rowType;
+    private String basePath;
+    private Configuration conf;
+    private DataPruner dataPruner;
+    private List<ExpressionEvaluators.Evaluator> partitionEvaluators;
+    private List<String> partitionKeys;
+    private List<DataType> partitionTypes;
+    private String defaultParName;
+    private boolean hivePartition;
+    private Collection<String> candidatePartitions;
+
+    private Builder() {
+    }
+
+    public Builder rowType(RowType rowType) {
+      this.rowType = rowType;
+      return this;
+    }
+
+    public Builder basePath(String basePath) {
+      this.basePath = basePath;
+      return this;
+    }
+
+    public Builder conf(Configuration conf) {
+      this.conf = conf;
+      return this;
+    }
+
+    public Builder dataPruner(DataPruner dataPruner) {
+      this.dataPruner = dataPruner;
+      return this;
+    }
+
+    public Builder partitionEvaluators(List<Evaluator> partitionEvaluators) {
+      this.partitionEvaluators = partitionEvaluators;
+      return this;
+    }
+
+    public Builder partitionKeys(List<String> partitionKeys) {
+      this.partitionKeys = partitionKeys;
+      return this;
+    }
+
+    public Builder partitionTypes(List<DataType> partitionTypes) {
+      this.partitionTypes = partitionTypes;
+      return this;
+    }
+
+    public Builder defaultParName(String defaultParName) {
+      this.defaultParName = defaultParName;
+      return this;
+    }
+
+    public Builder hivePartition(boolean hivePartition) {
+      this.hivePartition = hivePartition;
+      return this;
+    }
+
+    public Builder candidatePartitions(Collection<String> candidatePartitions) {
+      this.candidatePartitions = candidatePartitions;
+      return this;
+    }
+
+    public PartitionPruner build() {
+      PartitionPruner staticPruner = null;
+      if (candidatePartitions != null && !candidatePartitions.isEmpty()) {
+        staticPruner = new StaticPartitionPruner(candidatePartitions);
+      }
+      PartitionPruner dynamicPruner = null;
+      if (partitionEvaluators != null && !partitionEvaluators.isEmpty()) {
+        Preconditions.checkArgument(partitionKeys != null && partitionTypes != null && defaultParName != null);
+        dynamicPruner = new DynamicPartitionPruner(partitionEvaluators, partitionKeys, partitionTypes, defaultParName, hivePartition);
+      }
+      PartitionPruner columnStatsPruner = null;
+      if (dataPruner != null
+          && conf.get(FlinkOptions.READ_PARTITION_DATA_SKIPPING_ENABLED)
+          && conf.get(FlinkOptions.METADATA_ENABLED)) {
+        Preconditions.checkArgument(rowType != null && basePath != null && conf != null);
+        columnStatsPruner = new ColumnStatsPartitionPruner(rowType, basePath, StreamerUtil.metadataConfig(conf), dataPruner);
+      }
+      List<PartitionPruner> partitionPruners =

Review Comment:
   We should use specific pruner if it is singleton.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12132:
URL: https://github.com/apache/hudi/pull/12132#discussion_r1808050113


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/FileIndex.java:
##########
@@ -83,8 +77,8 @@ private FileIndex(StoragePath path, Configuration conf, RowType rowType, DataPru
     this.rowType = rowType;
     this.hadoopConf = HadoopConfigurations.getHadoopConf(conf);
     this.tableExists = StreamerUtil.tableExists(path.toString(), hadoopConf);
-    this.metadataConfig = metadataConfig(conf);
-    this.dataPruner = isDataSkippingFeasible(conf.getBoolean(FlinkOptions.READ_DATA_SKIPPING_ENABLED)) ? dataPruner : null;
+    this.metadataConfig = StreamerUtil.metadataConfig(conf);
+    this.dataPruner = isDataSkippingFeasible(conf.get(FlinkOptions.READ_DATA_SKIPPING_ENABLED)) ? dataPruner : null;

Review Comment:
   Maybe we should rename the pruner to `colStatsPruner` because it is used for partition pruning now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12132:
URL: https://github.com/apache/hudi/pull/12132#issuecomment-2436672565

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9bff28bbdd59593f5c50bdbfca7af57543abd4be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1296",
       "triggerID" : "9bff28bbdd59593f5c50bdbfca7af57543abd4be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5f7b6934482578550b592c30e015bdc7bafea52a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1350",
       "triggerID" : "5f7b6934482578550b592c30e015bdc7bafea52a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7b9a50506e0e16f49da45498b1dd79e99300ac3a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1353",
       "triggerID" : "7b9a50506e0e16f49da45498b1dd79e99300ac3a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "06a5ca03ecb59ae6fb9b33de62fdb1e30bd747c3",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1393",
       "triggerID" : "06a5ca03ecb59ae6fb9b33de62fdb1e30bd747c3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "efde4491a94d517528c04c94449be0aee37d262e",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1409",
       "triggerID" : "efde4491a94d517528c04c94449be0aee37d262e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 06a5ca03ecb59ae6fb9b33de62fdb1e30bd747c3 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1393) 
   * efde4491a94d517528c04c94449be0aee37d262e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1409) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12132:
URL: https://github.com/apache/hudi/pull/12132#issuecomment-2436738880

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9bff28bbdd59593f5c50bdbfca7af57543abd4be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1296",
       "triggerID" : "9bff28bbdd59593f5c50bdbfca7af57543abd4be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5f7b6934482578550b592c30e015bdc7bafea52a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1350",
       "triggerID" : "5f7b6934482578550b592c30e015bdc7bafea52a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7b9a50506e0e16f49da45498b1dd79e99300ac3a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1353",
       "triggerID" : "7b9a50506e0e16f49da45498b1dd79e99300ac3a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "06a5ca03ecb59ae6fb9b33de62fdb1e30bd747c3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1393",
       "triggerID" : "06a5ca03ecb59ae6fb9b33de62fdb1e30bd747c3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "efde4491a94d517528c04c94449be0aee37d262e",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1409",
       "triggerID" : "efde4491a94d517528c04c94449be0aee37d262e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * efde4491a94d517528c04c94449be0aee37d262e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1409) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12132:
URL: https://github.com/apache/hudi/pull/12132#discussion_r1814090420


##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/ITTestHoodieDataSource.java:
##########
@@ -507,6 +508,61 @@ void testStreamReadWithDeletes() throws Exception {
     assertRowsEquals(result, expected, true);
   }
 
+  @ParameterizedTest
+  @MethodSource("executionModeAndTableTypeParams")
+  void testReadWithPartitionStatsPrune(ExecMode execMode, HoodieTableType tableType) throws Exception {

Review Comment:
   instead of execution mode, can we test the combination of hiveStylePartitioning and table type.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12132:
URL: https://github.com/apache/hudi/pull/12132#issuecomment-2430684149

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9bff28bbdd59593f5c50bdbfca7af57543abd4be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1296",
       "triggerID" : "9bff28bbdd59593f5c50bdbfca7af57543abd4be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5f7b6934482578550b592c30e015bdc7bafea52a",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1350",
       "triggerID" : "5f7b6934482578550b592c30e015bdc7bafea52a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5f7b6934482578550b592c30e015bdc7bafea52a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1350) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12132:
URL: https://github.com/apache/hudi/pull/12132#discussion_r1814088689


##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/source/TestFileIndex.java:
##########
@@ -157,6 +160,47 @@ void testFileListingWithDataSkipping() throws Exception {
     assertThat(files.size(), is(2));
   }
 
+  @Test
+  void testFileListingWithPartitionStatsPruning() throws Exception {

Review Comment:
   can we test both cow and mor table.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12132:
URL: https://github.com/apache/hudi/pull/12132#issuecomment-2425443318

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9bff28bbdd59593f5c50bdbfca7af57543abd4be",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1296",
       "triggerID" : "9bff28bbdd59593f5c50bdbfca7af57543abd4be",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9bff28bbdd59593f5c50bdbfca7af57543abd4be Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1296) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12132:
URL: https://github.com/apache/hudi/pull/12132#discussion_r1808053870


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/stats/ColumnStatsIndices.java:
##########
@@ -60,16 +64,16 @@
 import static org.apache.hudi.common.util.ValidationUtils.checkState;
 
 /**
- * Utilities for abstracting away heavy-lifting of interactions with Metadata Table's Column Stats Index,
- * providing convenient interfaces to read it, transpose, etc.
+ * Utilities for abstracting away heavy-lifting of interactions with Metadata Table's Column Stats Index
+ * or Partition Stats Index, providing convenient interfaces to read it, transpose, etc.
  */
 public class ColumnStatsIndices {
   private static final DataType METADATA_DATA_TYPE = getMetadataDataType();
   private static final DataType COL_STATS_DATA_TYPE = getColStatsDataType();
   private static final int[] COL_STATS_TARGET_POS = getColStatsTargetPos();
 
   // the column schema:
-  // |- file_name: string
+  // |- file_name: string -- file name for column stats and partition name for partition stats

Review Comment:
   file_name -> target_name?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #12132:
URL: https://github.com/apache/hudi/pull/12132#discussion_r1808648628


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -378,6 +378,14 @@ private FlinkOptions() {
       .withDescription("Enables data-skipping allowing queries to leverage indexes to reduce the search space by "
           + "skipping over files");
 
+  @AdvancedConfig
+  public static final ConfigOption<Boolean> READ_PARTITION_DATA_SKIPPING_ENABLED = ConfigOptions
+      .key("read.partition.data.skipping.enabled")

Review Comment:
   The main purpose here is to provide the ability to enable different indexes separately, just in case that the loading of a specific index is quite time-consuming or there are some defects in the implementation of an index.  Maybe we can change the code as all indexes are enabled by default when data skipping is enabled, and introduce a new config option 'read.data.skipping.excluded.indexes' to disable some indexes if necessary. WDYT?



##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/FileIndex.java:
##########
@@ -83,8 +77,8 @@ private FileIndex(StoragePath path, Configuration conf, RowType rowType, DataPru
     this.rowType = rowType;
     this.hadoopConf = HadoopConfigurations.getHadoopConf(conf);
     this.tableExists = StreamerUtil.tableExists(path.toString(), hadoopConf);
-    this.metadataConfig = metadataConfig(conf);
-    this.dataPruner = isDataSkippingFeasible(conf.getBoolean(FlinkOptions.READ_DATA_SKIPPING_ENABLED)) ? dataPruner : null;
+    this.metadataConfig = StreamerUtil.metadataConfig(conf);
+    this.dataPruner = isDataSkippingFeasible(conf.get(FlinkOptions.READ_DATA_SKIPPING_ENABLED)) ? dataPruner : null;

Review Comment:
   Actually, the pruner here is still used for files pruning. The pruner for partition pruning is in `ColumnStatsPartitionPruner`, I can change the naming there.



##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/prune/PartitionPruners.java:
##########
@@ -125,6 +138,154 @@ public Set<String> filter(Collection<String> partitions) {
     }
   }
 
+  /**
+   * ColumnStats partition pruner for hoodie table source which enables partition stats index.
+   *
+   * <p>Note: The data of new partitions created after the job starts could be read if they match the
+   * filter conditions.
+   */
+  public static class ColumnStatsPartitionPruner implements PartitionPruner {
+    private final String basePath;
+    private final HoodieMetadataConfig metadataConfig;
+    private final DataPruner dataPruner;
+    private final RowType rowType;
+
+    public ColumnStatsPartitionPruner(
+        RowType rowType,
+        String basePath,
+        HoodieMetadataConfig metadataConfig,
+        DataPruner dataPruner) {
+      this.rowType = rowType;
+      this.basePath = basePath;
+      this.metadataConfig = metadataConfig;
+      this.dataPruner = dataPruner;
+    }
+
+    @Override
+    public Set<String> filter(Collection<String> partitions) {
+      Set<String> candidatePartitions = ColumnStatsIndices.candidatePartitionsInMetadataTable(
+          basePath, metadataConfig, rowType, dataPruner, new ArrayList<>(partitions));
+      if (candidatePartitions == null) {
+        return new HashSet<>(partitions);
+      }
+      return partitions.stream().filter(candidatePartitions::contains).collect(Collectors.toSet());
+    }
+  }
+
+  /**
+   * Chained partition pruner for hoodie table source which combines multiple partition pruners.
+   */
+  public static class ChainedPartitionPruner implements PartitionPruner {
+    private final List<PartitionPruner> pruners;
+
+    public ChainedPartitionPruner(List<PartitionPruner> pruners) {
+      this.pruners = pruners;
+    }
+
+    @Override
+    public Set<String> filter(Collection<String> partitions) {
+      for (PartitionPruner pruner: pruners) {
+        partitions = pruner.filter(partitions);
+      }
+      return new HashSet<>(partitions);
+    }
+  }
+
+  public static Builder builder() {
+    return new Builder();
+  }
+
+  public static class Builder {
+    private RowType rowType;
+    private String basePath;
+    private Configuration conf;
+    private DataPruner dataPruner;
+    private List<ExpressionEvaluators.Evaluator> partitionEvaluators;
+    private List<String> partitionKeys;
+    private List<DataType> partitionTypes;
+    private String defaultParName;
+    private boolean hivePartition;
+    private Collection<String> candidatePartitions;
+
+    private Builder() {
+    }
+
+    public Builder rowType(RowType rowType) {
+      this.rowType = rowType;
+      return this;
+    }
+
+    public Builder basePath(String basePath) {
+      this.basePath = basePath;
+      return this;
+    }
+
+    public Builder conf(Configuration conf) {
+      this.conf = conf;
+      return this;
+    }

Review Comment:
   Before the PR, there is only one type of partition pruner is needed in`HoodieTableSource`, as we introduce more partition pruners, there are some orchestration logics among the pruners. Maybe it's better to encapsulate those logics inside PartitionPruners, and builder allows more flexible creation of pruners. WDYT?



##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/stats/ColumnStatsIndices.java:
##########
@@ -60,16 +64,16 @@
 import static org.apache.hudi.common.util.ValidationUtils.checkState;
 
 /**
- * Utilities for abstracting away heavy-lifting of interactions with Metadata Table's Column Stats Index,
- * providing convenient interfaces to read it, transpose, etc.
+ * Utilities for abstracting away heavy-lifting of interactions with Metadata Table's Column Stats Index
+ * or Partition Stats Index, providing convenient interfaces to read it, transpose, etc.

Review Comment:
   LGTM



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12132:
URL: https://github.com/apache/hudi/pull/12132#issuecomment-2430804185

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9bff28bbdd59593f5c50bdbfca7af57543abd4be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1296",
       "triggerID" : "9bff28bbdd59593f5c50bdbfca7af57543abd4be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5f7b6934482578550b592c30e015bdc7bafea52a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1350",
       "triggerID" : "5f7b6934482578550b592c30e015bdc7bafea52a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7b9a50506e0e16f49da45498b1dd79e99300ac3a",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1353",
       "triggerID" : "7b9a50506e0e16f49da45498b1dd79e99300ac3a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 7b9a50506e0e16f49da45498b1dd79e99300ac3a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1353) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12132:
URL: https://github.com/apache/hudi/pull/12132#issuecomment-2430702319

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9bff28bbdd59593f5c50bdbfca7af57543abd4be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1296",
       "triggerID" : "9bff28bbdd59593f5c50bdbfca7af57543abd4be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5f7b6934482578550b592c30e015bdc7bafea52a",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1350",
       "triggerID" : "5f7b6934482578550b592c30e015bdc7bafea52a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7b9a50506e0e16f49da45498b1dd79e99300ac3a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7b9a50506e0e16f49da45498b1dd79e99300ac3a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5f7b6934482578550b592c30e015bdc7bafea52a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1350) 
   * 7b9a50506e0e16f49da45498b1dd79e99300ac3a UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12132:
URL: https://github.com/apache/hudi/pull/12132#issuecomment-2425442238

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9bff28bbdd59593f5c50bdbfca7af57543abd4be",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9bff28bbdd59593f5c50bdbfca7af57543abd4be",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9bff28bbdd59593f5c50bdbfca7af57543abd4be UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12132:
URL: https://github.com/apache/hudi/pull/12132#discussion_r1808052401


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/prune/PartitionPruners.java:
##########
@@ -125,6 +138,154 @@ public Set<String> filter(Collection<String> partitions) {
     }
   }
 
+  /**
+   * ColumnStats partition pruner for hoodie table source which enables partition stats index.
+   *
+   * <p>Note: The data of new partitions created after the job starts could be read if they match the
+   * filter conditions.
+   */
+  public static class ColumnStatsPartitionPruner implements PartitionPruner {
+    private final String basePath;
+    private final HoodieMetadataConfig metadataConfig;
+    private final DataPruner dataPruner;
+    private final RowType rowType;
+
+    public ColumnStatsPartitionPruner(
+        RowType rowType,
+        String basePath,
+        HoodieMetadataConfig metadataConfig,
+        DataPruner dataPruner) {
+      this.rowType = rowType;
+      this.basePath = basePath;
+      this.metadataConfig = metadataConfig;
+      this.dataPruner = dataPruner;
+    }
+
+    @Override
+    public Set<String> filter(Collection<String> partitions) {
+      Set<String> candidatePartitions = ColumnStatsIndices.candidatePartitionsInMetadataTable(
+          basePath, metadataConfig, rowType, dataPruner, new ArrayList<>(partitions));
+      if (candidatePartitions == null) {
+        return new HashSet<>(partitions);
+      }
+      return partitions.stream().filter(candidatePartitions::contains).collect(Collectors.toSet());
+    }
+  }
+
+  /**
+   * Chained partition pruner for hoodie table source which combines multiple partition pruners.
+   */
+  public static class ChainedPartitionPruner implements PartitionPruner {
+    private final List<PartitionPruner> pruners;
+
+    public ChainedPartitionPruner(List<PartitionPruner> pruners) {
+      this.pruners = pruners;
+    }
+
+    @Override
+    public Set<String> filter(Collection<String> partitions) {
+      for (PartitionPruner pruner: pruners) {
+        partitions = pruner.filter(partitions);
+      }
+      return new HashSet<>(partitions);
+    }
+  }
+
+  public static Builder builder() {
+    return new Builder();
+  }
+
+  public static class Builder {
+    private RowType rowType;
+    private String basePath;
+    private Configuration conf;
+    private DataPruner dataPruner;
+    private List<ExpressionEvaluators.Evaluator> partitionEvaluators;
+    private List<String> partitionKeys;
+    private List<DataType> partitionTypes;
+    private String defaultParName;
+    private boolean hivePartition;
+    private Collection<String> candidatePartitions;
+
+    private Builder() {
+    }
+
+    public Builder rowType(RowType rowType) {
+      this.rowType = rowType;
+      return this;
+    }
+
+    public Builder basePath(String basePath) {
+      this.basePath = basePath;
+      return this;
+    }
+
+    public Builder conf(Configuration conf) {
+      this.conf = conf;
+      return this;
+    }
+
+    public Builder dataPruner(DataPruner dataPruner) {
+      this.dataPruner = dataPruner;
+      return this;
+    }
+
+    public Builder partitionEvaluators(List<Evaluator> partitionEvaluators) {
+      this.partitionEvaluators = partitionEvaluators;
+      return this;
+    }
+
+    public Builder partitionKeys(List<String> partitionKeys) {
+      this.partitionKeys = partitionKeys;
+      return this;
+    }
+
+    public Builder partitionTypes(List<DataType> partitionTypes) {
+      this.partitionTypes = partitionTypes;
+      return this;
+    }
+
+    public Builder defaultParName(String defaultParName) {
+      this.defaultParName = defaultParName;
+      return this;
+    }
+
+    public Builder hivePartition(boolean hivePartition) {
+      this.hivePartition = hivePartition;
+      return this;
+    }
+
+    public Builder candidatePartitions(Collection<String> candidatePartitions) {
+      this.candidatePartitions = candidatePartitions;
+      return this;
+    }
+
+    public PartitionPruner build() {
+      PartitionPruner staticPruner = null;
+      if (candidatePartitions != null && !candidatePartitions.isEmpty()) {
+        staticPruner = new StaticPartitionPruner(candidatePartitions);
+      }
+      PartitionPruner dynamicPruner = null;
+      if (partitionEvaluators != null && !partitionEvaluators.isEmpty()) {
+        Preconditions.checkArgument(partitionKeys != null && partitionTypes != null && defaultParName != null);

Review Comment:
   Add user friendly warnings msgs for these check always.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12132:
URL: https://github.com/apache/hudi/pull/12132#discussion_r1808054160


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/stats/ColumnStatsIndices.java:
##########
@@ -60,16 +64,16 @@
 import static org.apache.hudi.common.util.ValidationUtils.checkState;
 
 /**
- * Utilities for abstracting away heavy-lifting of interactions with Metadata Table's Column Stats Index,
- * providing convenient interfaces to read it, transpose, etc.
+ * Utilities for abstracting away heavy-lifting of interactions with Metadata Table's Column Stats Index
+ * or Partition Stats Index, providing convenient interfaces to read it, transpose, etc.

Review Comment:
   The doc has inconsistency with the class name.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #12132:
URL: https://github.com/apache/hudi/pull/12132#discussion_r1808648628


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -378,6 +378,14 @@ private FlinkOptions() {
       .withDescription("Enables data-skipping allowing queries to leverage indexes to reduce the search space by "
           + "skipping over files");
 
+  @AdvancedConfig
+  public static final ConfigOption<Boolean> READ_PARTITION_DATA_SKIPPING_ENABLED = ConfigOptions
+      .key("read.partition.data.skipping.enabled")

Review Comment:
   The main purpose here is to provide the ability to enable different indexes separately, just in case that there are unexpected behaviors in some indexes.  Maybe we can change the code as all indexes are enabled by default when data skipping is enabled, and introduce a new config option 'read.data.skipping.excluded.indexes' to disable some indexes if necessary. WDYT?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12132:
URL: https://github.com/apache/hudi/pull/12132#discussion_r1808048611


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -378,6 +378,14 @@ private FlinkOptions() {
       .withDescription("Enables data-skipping allowing queries to leverage indexes to reduce the search space by "
           + "skipping over files");
 
+  @AdvancedConfig
+  public static final ConfigOption<Boolean> READ_PARTITION_DATA_SKIPPING_ENABLED = ConfigOptions
+      .key("read.partition.data.skipping.enabled")

Review Comment:
   This option can be eliminated, when data skipping is enabled, we always enable partiton stats pruning if the partition metadata is there.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12132:
URL: https://github.com/apache/hudi/pull/12132#discussion_r1808052131


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/prune/PartitionPruners.java:
##########
@@ -125,6 +138,154 @@ public Set<String> filter(Collection<String> partitions) {
     }
   }
 
+  /**
+   * ColumnStats partition pruner for hoodie table source which enables partition stats index.
+   *
+   * <p>Note: The data of new partitions created after the job starts could be read if they match the
+   * filter conditions.
+   */
+  public static class ColumnStatsPartitionPruner implements PartitionPruner {
+    private final String basePath;
+    private final HoodieMetadataConfig metadataConfig;
+    private final DataPruner dataPruner;
+    private final RowType rowType;
+
+    public ColumnStatsPartitionPruner(
+        RowType rowType,
+        String basePath,
+        HoodieMetadataConfig metadataConfig,
+        DataPruner dataPruner) {
+      this.rowType = rowType;
+      this.basePath = basePath;
+      this.metadataConfig = metadataConfig;
+      this.dataPruner = dataPruner;
+    }
+
+    @Override
+    public Set<String> filter(Collection<String> partitions) {
+      Set<String> candidatePartitions = ColumnStatsIndices.candidatePartitionsInMetadataTable(
+          basePath, metadataConfig, rowType, dataPruner, new ArrayList<>(partitions));
+      if (candidatePartitions == null) {
+        return new HashSet<>(partitions);
+      }
+      return partitions.stream().filter(candidatePartitions::contains).collect(Collectors.toSet());
+    }
+  }
+
+  /**
+   * Chained partition pruner for hoodie table source which combines multiple partition pruners.
+   */
+  public static class ChainedPartitionPruner implements PartitionPruner {
+    private final List<PartitionPruner> pruners;
+
+    public ChainedPartitionPruner(List<PartitionPruner> pruners) {
+      this.pruners = pruners;
+    }
+
+    @Override
+    public Set<String> filter(Collection<String> partitions) {
+      for (PartitionPruner pruner: pruners) {
+        partitions = pruner.filter(partitions);
+      }
+      return new HashSet<>(partitions);
+    }
+  }
+
+  public static Builder builder() {
+    return new Builder();
+  }
+
+  public static class Builder {
+    private RowType rowType;
+    private String basePath;
+    private Configuration conf;
+    private DataPruner dataPruner;
+    private List<ExpressionEvaluators.Evaluator> partitionEvaluators;
+    private List<String> partitionKeys;
+    private List<DataType> partitionTypes;
+    private String defaultParName;
+    private boolean hivePartition;
+    private Collection<String> candidatePartitions;
+
+    private Builder() {
+    }
+
+    public Builder rowType(RowType rowType) {
+      this.rowType = rowType;
+      return this;
+    }
+
+    public Builder basePath(String basePath) {
+      this.basePath = basePath;
+      return this;
+    }
+
+    public Builder conf(Configuration conf) {
+      this.conf = conf;
+      return this;
+    }

Review Comment:
   should we use builder here? The different pruners have disparity in parameters.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12132:
URL: https://github.com/apache/hudi/pull/12132#discussion_r1808053870


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/stats/ColumnStatsIndices.java:
##########
@@ -60,16 +64,16 @@
 import static org.apache.hudi.common.util.ValidationUtils.checkState;
 
 /**
- * Utilities for abstracting away heavy-lifting of interactions with Metadata Table's Column Stats Index,
- * providing convenient interfaces to read it, transpose, etc.
+ * Utilities for abstracting away heavy-lifting of interactions with Metadata Table's Column Stats Index
+ * or Partition Stats Index, providing convenient interfaces to read it, transpose, etc.
  */
 public class ColumnStatsIndices {
   private static final DataType METADATA_DATA_TYPE = getMetadataDataType();
   private static final DataType COL_STATS_DATA_TYPE = getColStatsDataType();
   private static final int[] COL_STATS_TARGET_POS = getColStatsTargetPos();
 
   // the column schema:
-  // |- file_name: string
+  // |- file_name: string -- file name for column stats and partition name for partition stats

Review Comment:
   file_name -> idx_key ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12132:
URL: https://github.com/apache/hudi/pull/12132#issuecomment-2430595465

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9bff28bbdd59593f5c50bdbfca7af57543abd4be",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1296",
       "triggerID" : "9bff28bbdd59593f5c50bdbfca7af57543abd4be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5f7b6934482578550b592c30e015bdc7bafea52a",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1350",
       "triggerID" : "5f7b6934482578550b592c30e015bdc7bafea52a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9bff28bbdd59593f5c50bdbfca7af57543abd4be Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1296) 
   * 5f7b6934482578550b592c30e015bdc7bafea52a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1350) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12132:
URL: https://github.com/apache/hudi/pull/12132#issuecomment-2435463640

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9bff28bbdd59593f5c50bdbfca7af57543abd4be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1296",
       "triggerID" : "9bff28bbdd59593f5c50bdbfca7af57543abd4be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5f7b6934482578550b592c30e015bdc7bafea52a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1350",
       "triggerID" : "5f7b6934482578550b592c30e015bdc7bafea52a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7b9a50506e0e16f49da45498b1dd79e99300ac3a",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1353",
       "triggerID" : "7b9a50506e0e16f49da45498b1dd79e99300ac3a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "06a5ca03ecb59ae6fb9b33de62fdb1e30bd747c3",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1393",
       "triggerID" : "06a5ca03ecb59ae6fb9b33de62fdb1e30bd747c3",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 7b9a50506e0e16f49da45498b1dd79e99300ac3a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1353) 
   * 06a5ca03ecb59ae6fb9b33de62fdb1e30bd747c3 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1393) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12132:
URL: https://github.com/apache/hudi/pull/12132#issuecomment-2435668487

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9bff28bbdd59593f5c50bdbfca7af57543abd4be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1296",
       "triggerID" : "9bff28bbdd59593f5c50bdbfca7af57543abd4be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5f7b6934482578550b592c30e015bdc7bafea52a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1350",
       "triggerID" : "5f7b6934482578550b592c30e015bdc7bafea52a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7b9a50506e0e16f49da45498b1dd79e99300ac3a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1353",
       "triggerID" : "7b9a50506e0e16f49da45498b1dd79e99300ac3a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "06a5ca03ecb59ae6fb9b33de62fdb1e30bd747c3",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1393",
       "triggerID" : "06a5ca03ecb59ae6fb9b33de62fdb1e30bd747c3",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 06a5ca03ecb59ae6fb9b33de62fdb1e30bd747c3 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1393) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12132:
URL: https://github.com/apache/hudi/pull/12132#discussion_r1808054160


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/stats/ColumnStatsIndices.java:
##########
@@ -60,16 +64,16 @@
 import static org.apache.hudi.common.util.ValidationUtils.checkState;
 
 /**
- * Utilities for abstracting away heavy-lifting of interactions with Metadata Table's Column Stats Index,
- * providing convenient interfaces to read it, transpose, etc.
+ * Utilities for abstracting away heavy-lifting of interactions with Metadata Table's Column Stats Index
+ * or Partition Stats Index, providing convenient interfaces to read it, transpose, etc.

Review Comment:
   The doc has inconsistency with the class name. Maybe we just keep the doc as it is if we agree both `file stats` and `partition stats` belong to `column stats`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12132:
URL: https://github.com/apache/hudi/pull/12132#discussion_r1809727026


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/FileIndex.java:
##########
@@ -177,7 +174,8 @@ public List<StoragePathInfo> getFilesInPartitions() {
     }
 
     // data skipping
-    Set<String> candidateFiles = candidateFilesInMetadataTable(allFiles);
+    Set<String> candidateFiles = ColumnStatsIndices.candidateFilesInMetadataTable(path.toString(), metadataConfig,
+        rowType, dataPruner, allFiles.stream().map(StoragePathInfo::toString).collect(Collectors.toList()));

Review Comment:
   `StoragePathInfo::toString` returns full path instead, we use file name match in metadata table before.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12132:
URL: https://github.com/apache/hudi/pull/12132#issuecomment-2425532149

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9bff28bbdd59593f5c50bdbfca7af57543abd4be",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1296",
       "triggerID" : "9bff28bbdd59593f5c50bdbfca7af57543abd4be",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9bff28bbdd59593f5c50bdbfca7af57543abd4be Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1296) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #12132:
URL: https://github.com/apache/hudi/pull/12132


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12132:
URL: https://github.com/apache/hudi/pull/12132#issuecomment-2435459450

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9bff28bbdd59593f5c50bdbfca7af57543abd4be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1296",
       "triggerID" : "9bff28bbdd59593f5c50bdbfca7af57543abd4be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5f7b6934482578550b592c30e015bdc7bafea52a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1350",
       "triggerID" : "5f7b6934482578550b592c30e015bdc7bafea52a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7b9a50506e0e16f49da45498b1dd79e99300ac3a",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1353",
       "triggerID" : "7b9a50506e0e16f49da45498b1dd79e99300ac3a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "06a5ca03ecb59ae6fb9b33de62fdb1e30bd747c3",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "06a5ca03ecb59ae6fb9b33de62fdb1e30bd747c3",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 7b9a50506e0e16f49da45498b1dd79e99300ac3a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1353) 
   * 06a5ca03ecb59ae6fb9b33de62fdb1e30bd747c3 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #12132:
URL: https://github.com/apache/hudi/pull/12132#discussion_r1810177941


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/FileIndex.java:
##########
@@ -177,7 +174,8 @@ public List<StoragePathInfo> getFilesInPartitions() {
     }
 
     // data skipping
-    Set<String> candidateFiles = candidateFilesInMetadataTable(allFiles);
+    Set<String> candidateFiles = ColumnStatsIndices.candidateFilesInMetadataTable(path.toString(), metadataConfig,
+        rowType, dataPruner, allFiles.stream().map(StoragePathInfo::toString).collect(Collectors.toList()));

Review Comment:
   Nice catch!



##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -378,6 +378,14 @@ private FlinkOptions() {
       .withDescription("Enables data-skipping allowing queries to leverage indexes to reduce the search space by "
           + "skipping over files");
 
+  @AdvancedConfig
+  public static final ConfigOption<Boolean> READ_PARTITION_DATA_SKIPPING_ENABLED = ConfigOptions
+      .key("read.partition.data.skipping.enabled")

Review Comment:
   ok, I'll update.



##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/stats/ColumnStatsIndices.java:
##########
@@ -85,16 +89,109 @@ public class ColumnStatsIndices {
   private ColumnStatsIndices() {
   }
 
-  public static List<RowData> readColumnStatsIndex(String basePath, HoodieMetadataConfig metadataConfig, String[] targetColumns) {
+  public static Set<String> candidatePartitionsInMetadataTable(

Review Comment:
   I did not make abstracts in the PR because Partition Stats Index and Column Stats Index share same API and same storage schema in metadata table. But it sounds ok to have some basic abstractions here as we plan to introduce more indexes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12132:
URL: https://github.com/apache/hudi/pull/12132#issuecomment-2430704688

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9bff28bbdd59593f5c50bdbfca7af57543abd4be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1296",
       "triggerID" : "9bff28bbdd59593f5c50bdbfca7af57543abd4be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5f7b6934482578550b592c30e015bdc7bafea52a",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1350",
       "triggerID" : "5f7b6934482578550b592c30e015bdc7bafea52a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7b9a50506e0e16f49da45498b1dd79e99300ac3a",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1353",
       "triggerID" : "7b9a50506e0e16f49da45498b1dd79e99300ac3a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5f7b6934482578550b592c30e015bdc7bafea52a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1350) 
   * 7b9a50506e0e16f49da45498b1dd79e99300ac3a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1353) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8196] Support pruning based on partition stats index in Hudi Flink [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12132:
URL: https://github.com/apache/hudi/pull/12132#issuecomment-2430594056

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9bff28bbdd59593f5c50bdbfca7af57543abd4be",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1296",
       "triggerID" : "9bff28bbdd59593f5c50bdbfca7af57543abd4be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5f7b6934482578550b592c30e015bdc7bafea52a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "5f7b6934482578550b592c30e015bdc7bafea52a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9bff28bbdd59593f5c50bdbfca7af57543abd4be Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1296) 
   * 5f7b6934482578550b592c30e015bdc7bafea52a UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org