You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "danny0405 (via GitHub)" <gi...@apache.org> on 2026/09/18 03:08:35 UTC

[PR] perf(flink): scope column stats reads to candidate partitions [hudi]

danny0405 opened a new pull request, #19992:
URL: https://github.com/apache/hudi/pull/19992

   ### Describe the issue this Pull Request addresses
   
   Flink reads column statistics using column-only key prefixes even after partition, bucket, or record-index pruning has narrowed the candidate files. A query selecting one partition can therefore fetch and transpose statistics across the table.
   
   ### Summary and Changelog
   
   Scope column-stat reads to the partitions containing the remaining file slices.
   
   - Pass distinct candidate partition paths from `FileIndex` into the column-stat reader after bucket and record-index pruning.
   - Build `ColumnStatsIndexPrefixRawKey(column, partition)` prefixes for those partitions. Use a non-null empty list for column-only reads, including partition-stat lookups.
   - Return before constructing filenames when no column-stat probe exists or no file slices remain.
   - Preserve conservative inclusion of files missing statistics.
   - Cover single and multiple partitions, Hive-style and nonpartitioned tables, duplicate partition paths, missing statistics, and the no-probe path in tests.
   
   No code was copied.
   
   ### Impact
   
   Reduces metadata reads and statistics transposition for Flink queries whose candidate files span only part of the table. Filtering results remain unchanged. The change updates Flink-internal statistics method signatures; it introduces no supported user API, configuration, or storage-format changes.
   
   ### Risk Level
   
   low
   
   Incorrect prefix scoping could omit statistics needed for pruning. Tests cover partition naming variants and retain candidates without statistics; existing partition-stat and record-index tests also pass.
   
   Validation:
   - `mvn -o -pl hudi-flink-datasource/hudi-flink -Dtest=TestColumnStatsIndex,TestFileIndex -Dsurefire.rerunFailingTestsCount=0 test`: 49 tests passed.
   - `mvn -o -pl hudi-flink-datasource/hudi-flink -Dtest=TestColumnStatsIndex -Dsurefire.rerunFailingTestsCount=0 test`: 5 tests passed after the final prefix-builder simplification.
   - Compilation, Checkstyle, Apache RAT, and `git diff --check` passed.
   
   ### Documentation Update
   
   none — this is an internal query-planning optimization with no new configuration or user workflow.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(flink): scope column stats reads to candidate partitions [hudi]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19992:
URL: https://github.com/apache/hudi/pull/19992#issuecomment-5724675778

   ## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19992?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 80.23%. Comparing base ([`e71e6cc`](https://app.codecov.io/gh/apache/hudi/commit/e71e6cc3f07c753b565fcf7739e14c12850f3733?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`27896d3`](https://app.codecov.io/gh/apache/hudi/commit/27896d33f76f23cce3910940698de46f83a50d37?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 4 commits behind head on master.
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #19992      +/-   ##
   ============================================
   - Coverage     80.33%   80.23%   -0.11%     
   - Complexity    34752    34835      +83     
   ============================================
     Files          2543     2543              
     Lines        142478   142937     +459     
     Branches      17308    17444     +136     
   ============================================
   + Hits         114464   114684     +220     
   - Misses        20119    20321     +202     
   - Partials       7895     7932      +37     
   ```
   
   | [Components](https://app.codecov.io/gh/apache/hudi/pull/19992/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/19992/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.38% <ø> (-0.47%)` | :arrow_down: |
   | [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/19992/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.36% <ø> (-0.01%)` | :arrow_down: |
   | [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/19992/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.69% <ø> (-0.01%)` | :arrow_down: |
   | [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/19992/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.77% <ø> (ø)` | |
   | [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/19992/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `78.20% <ø> (ø)` | |
   | [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/19992/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `69.99% <ø> (ø)` | |
   | [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/19992/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.91% <ø> (ø)` | |
   | [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/19992/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `76.02% <ø> (ø)` | |
   | [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/19992/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.61% <ø> (ø)` | |
   | [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/19992/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.34% <ø> (ø)` | |
   | [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/19992/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `80.99% <ø> (ø)` | |
   | [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/19992/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.20% <ø> (ø)` | |
   
   | [Flag](https://app.codecov.io/gh/apache/hudi/pull/19992/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19992/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.91% <ø> (+<0.01%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19992?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [...rc/main/java/org/apache/hudi/source/FileIndex.java](https://app.codecov.io/gh/apache/hudi/pull/19992?src=pr&el=tree&filepath=hudi-flink-datasource%2Fhudi-flink%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fsource%2FFileIndex.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1mbGluay1kYXRhc291cmNlL2h1ZGktZmxpbmsvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvc291cmNlL0ZpbGVJbmRleC5qYXZh) | `86.40% <ø> (-0.14%)` | :arrow_down: |
   | [...a/org/apache/hudi/source/stats/FileStatsIndex.java](https://app.codecov.io/gh/apache/hudi/pull/19992?src=pr&el=tree&filepath=hudi-flink-datasource%2Fhudi-flink%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fsource%2Fstats%2FFileStatsIndex.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1mbGluay1kYXRhc291cmNlL2h1ZGktZmxpbmsvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvc291cmNlL3N0YXRzL0ZpbGVTdGF0c0luZGV4LmphdmE=) | `83.94% <ø> (-0.46%)` | :arrow_down: |
   | [.../apache/hudi/source/stats/PartitionStatsIndex.java](https://app.codecov.io/gh/apache/hudi/pull/19992?src=pr&el=tree&filepath=hudi-flink-datasource%2Fhudi-flink%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fsource%2Fstats%2FPartitionStatsIndex.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1mbGluay1kYXRhc291cmNlL2h1ZGktZmxpbmsvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvc291cmNlL3N0YXRzL1BhcnRpdGlvblN0YXRzSW5kZXguamF2YQ==) | `66.66% <ø> (-4.77%)` | :arrow_down: |
   
   ... and [3 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19992/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   </details>
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(flink): scope column stats reads to candidate partitions [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19992:
URL: https://github.com/apache/hudi/pull/19992#issuecomment-5724596304

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "27896d33f76f23cce3910940698de46f83a50d37",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "27896d33f76f23cce3910940698de46f83a50d37",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 27896d33f76f23cce3910940698de46f83a50d37 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(flink): scope column stats reads to candidate partitions [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19992:
URL: https://github.com/apache/hudi/pull/19992#issuecomment-5724790911

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "27896d33f76f23cce3910940698de46f83a50d37",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17189",
       "triggerID" : "27896d33f76f23cce3910940698de46f83a50d37",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 27896d33f76f23cce3910940698de46f83a50d37 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17189) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(flink): scope column stats reads to candidate partitions [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19992:
URL: https://github.com/apache/hudi/pull/19992#issuecomment-5725464248

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "27896d33f76f23cce3910940698de46f83a50d37",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17189",
       "triggerID" : "27896d33f76f23cce3910940698de46f83a50d37",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 27896d33f76f23cce3910940698de46f83a50d37 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17189) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(flink): scope column stats reads to candidate partitions [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19992:
URL: https://github.com/apache/hudi/pull/19992#issuecomment-5739731085

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "27896d33f76f23cce3910940698de46f83a50d37",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17189",
       "triggerID" : "27896d33f76f23cce3910940698de46f83a50d37",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a9b969f81207fff6fc9c789a0bd8110d72853e9",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17203",
       "triggerID" : "6a9b969f81207fff6fc9c789a0bd8110d72853e9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 6a9b969f81207fff6fc9c789a0bd8110d72853e9 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17203) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(flink): scope column stats reads to candidate partitions [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19992:
URL: https://github.com/apache/hudi/pull/19992#issuecomment-5739125766

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "27896d33f76f23cce3910940698de46f83a50d37",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17189",
       "triggerID" : "27896d33f76f23cce3910940698de46f83a50d37",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a9b969f81207fff6fc9c789a0bd8110d72853e9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6a9b969f81207fff6fc9c789a0bd8110d72853e9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 27896d33f76f23cce3910940698de46f83a50d37 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17189) 
   * 6a9b969f81207fff6fc9c789a0bd8110d72853e9 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(flink): scope column stats reads to candidate partitions [hudi]

Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19992:
URL: https://github.com/apache/hudi/pull/19992#discussion_r4043854873


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/stats/FileStatsIndex.java:
##########
@@ -384,20 +384,26 @@ private static Object doUnpack(
     return converter.convert(rawVal);
   }
 
+  /**
+   * Reads statistics for the requested columns and relative partition paths.

Review Comment:
   🤖 nit: the original comment explained why fetching by column-name prefixes considerably trims down the data read; that rationale got dropped when this javadoc was reworded to only mention the partition-scoping behavior. Worth keeping both points since neither is obvious to a future reader.
   
   <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(flink): scope column stats reads to candidate partitions [hudi]

Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19992:
URL: https://github.com/apache/hudi/pull/19992#discussion_r4056061660


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/FileIndex.java:
##########
@@ -197,8 +198,16 @@ public List<FileSlice> filterFileSlices(List<FileSlice> fileSlices) {
     }
 
     // data skipping based on column stats
+    if (colStatsProbe == null || filteredFileSlices.isEmpty()) {
+      return filteredFileSlices;
+    }
+    List<String> candidatePartitions = filteredFileSlices.stream().map(FileSlice::getPartitionPath).distinct().collect(Collectors.toList());
+    // Avoid expanding column prefixes when the remaining files still span every partition.
+    if (candidatePartitions.size() == totalPartitionCount) {
+      candidatePartitions = Collections.emptyList();

Review Comment:
   🤖 nit: using an empty list to mean "all partitions" is a bit of a hidden sentinel here — have you considered a small named constant (e.g. `ALL_PARTITIONS = Collections.emptyList()`) or a short comment on this line so readers don't mistake it for "no partitions"?
   
   <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(flink): scope column stats reads to candidate partitions [hudi]

Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #19992:
URL: https://github.com/apache/hudi/pull/19992#discussion_r4045165209


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/stats/FileStatsIndex.java:
##########
@@ -384,20 +384,26 @@ private static Object doUnpack(
     return converter.convert(rawVal);
   }
 
+  /**
+   * Reads statistics for the requested columns and relative partition paths.
+   * An empty partition list reads all partitions using column-only prefixes.
+   */
   @VisibleForTesting
-  public List<RowData> readColumnStatsIndexByColumns(String[] targetColumns) {
-    // NOTE: If specific columns have been provided, we can considerably trim down amount of data fetched
-    //       by only fetching Column Stats Index records pertaining to the requested columns.
-    //       Otherwise, we fall back to read whole Column Stats Index
+  public List<RowData> readColumnStatsIndexByColumns(String[] targetColumns, List<String> candidatePartitions) {
     ValidationUtils.checkArgument(targetColumns.length > 0,
         "Column stats is only valid when push down filters have referenced columns");
 
     // Read Metadata Table's column stats Flink's RowData list by
-    //    - Fetching the records by key-prefixes (column names)
+    //    - Fetching the records by key-prefixes (column names and candidate partitions, when provided)
     //    - Deserializing fetched records into [[RowData]]s
-    List<ColumnStatsIndexPrefixRawKey> rawKeys = Arrays.stream(targetColumns)
-        .map(ColumnStatsIndexPrefixRawKey::new)  // Just column name, no partition
-        .collect(Collectors.toList());
+    List<ColumnStatsIndexPrefixRawKey> rawKeys;
+    if (candidatePartitions.isEmpty()) {
+      rawKeys = Arrays.stream(targetColumns).map(ColumnStatsIndexPrefixRawKey::new).collect(Collectors.toList());
+    } else {
+      rawKeys = candidatePartitions.stream().distinct()
+          .flatMap(partition -> Arrays.stream(targetColumns).map(column -> new ColumnStatsIndexPrefixRawKey(column, partition)))
+          .collect(Collectors.toList());

Review Comment:
   One concern for queries whose candidate files still span all partitions: this expands the lookup from C column prefixes to P × C partition-column prefixes without reducing the statistics returned. Although the prefixes are sorted and the reader can advance through the keys, the additional prefix encoding, sorting, and lookup processing could increase query-planning overhead for highly partitioned tables.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(flink): scope column stats reads to candidate partitions [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19992:
URL: https://github.com/apache/hudi/pull/19992#issuecomment-5739165092

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "27896d33f76f23cce3910940698de46f83a50d37",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17189",
       "triggerID" : "27896d33f76f23cce3910940698de46f83a50d37",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a9b969f81207fff6fc9c789a0bd8110d72853e9",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17203",
       "triggerID" : "6a9b969f81207fff6fc9c789a0bd8110d72853e9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 27896d33f76f23cce3910940698de46f83a50d37 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17189) 
   * 6a9b969f81207fff6fc9c789a0bd8110d72853e9 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17203) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(flink): scope column stats reads to candidate partitions [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #19992:
URL: https://github.com/apache/hudi/pull/19992


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org