You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2026/10/09 07:52:27 UTC
[PR] fix(spark): keep bootstrap skeleton files and partitions as data skipping candidates [hudi]
yihua opened a new pull request, #20267:
URL: https://github.com/apache/hudi/pull/20267
### Describe the issue this Pull Request addresses
closes #20266
part of #20257
On a METADATA_ONLY bootstrapped table with column stats enabled (the Spark default for later writes), the column stats of a bootstrapped file group come from its skeleton file, which only has the meta columns. Partition stats pruning then drops bootstrapped partitions, and column stats pruning drops skeleton files when the filter also references a meta column, so data-column filters with data skipping on return too few rows.
Partition stats pruning came from #10352 and #11556 (1.0.0), neither of which considered bootstrap skeleton files. Filling null stats for a file missing a column came from #5746; #10389 made that safe for columns unindexed across the whole table, which does not cover a single file missing an indexed column. Column stats became on by default for Spark writers in #12596 (1.0.1), which made this reachable with default settings.
### Summary and Changelog
Column stats and partition stats pruning no longer drop bootstrapped data they have no data-column stats for.
- `PartitionStatsIndexSupport.prunePartitions` does not prune partitions when the table has a bootstrap base path.
- `ColumnStatsIndexSupport.computeCandidateFileNames` keeps base files that have a bootstrap base file (skeleton files) as candidates, only on tables with a bootstrap base path.
- Test: `TestDataSourceForBootstrap#testMetadataBootstrapDataSkippingWithColumnStats` (COW and MOR, partition stats on and off) bootstraps, enables column stats on later writes, and checks that seven filters return the same rows with data skipping on and off. All four cases fail on master (e.g. `driver >= 'driver_5'` returns 18 rows instead of 56).
The companion bootstrap fixes (column stats initialization failing after a METADATA_ONLY bootstrap, and the record index on bootstrap tables) follow in separate PRs; the column stats initialization one should land after this, since it makes this path reachable right after the bootstrap. This PR touches `ColumnStatsIndexSupport` next to #20096, in different hunks.
### Impact
Fixes silent wrong results for data skipping on bootstrapped tables. Bootstrapped tables (including FULL_RECORD ones) no longer use partition stats for pruning, and skeleton files are never skipped by column stats; tables without a bootstrap base path are unchanged and do no extra work. Indexing the bootstrap source files' stats would restore that pruning and can follow separately.
Known remaining gap, not a regression: the expression index on a bootstrapped table is also built from the skeleton file (`SparkMetadataWriterUtils` reads `new HoodieBaseFile(filePath)` without its bootstrap source), and with partition stats pruning now skipped such tables reach expression index partition pruning in `HoodieFileIndex`; this is not addressed here. Data skipping for `hoodie.bootstrap.data.queries.only` reads is handled by a separate fix.
### Risk Level
low: the change only removes pruning on tables with a bootstrap base path, so it can read more files there but cannot drop rows.
### Documentation Update
none
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): keep bootstrap skeleton files and partitions as data skipping candidates [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #20267:
URL: https://github.com/apache/hudi/pull/20267#issuecomment-6077500819
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/20267?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `22.22222%` with `7 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 76.49%. Comparing base ([`308f78e`](https://app.codecov.io/gh/apache/hudi/commit/308f78e71deabf7a9eec4b0a2ea26b1360658030?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`de7cbec`](https://app.codecov.io/gh/apache/hudi/commit/de7cbec8730a502606c3ffe05e1e05c3f4ab6bbd?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/20267?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...cala/org/apache/hudi/ColumnStatsIndexSupport.scala](https://app.codecov.io/gh/apache/hudi/pull/20267?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fhudi%2FColumnStatsIndexSupport.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvaHVkaS9Db2x1bW5TdGF0c0luZGV4U3VwcG9ydC5zY2FsYQ==) | 25.00% | [5 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/20267?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...a/org/apache/hudi/PartitionStatsIndexSupport.scala](https://app.codecov.io/gh/apache/hudi/pull/20267?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fhudi%2FPartitionStatsIndexSupport.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvaHVkaS9QYXJ0aXRpb25TdGF0c0luZGV4U3VwcG9ydC5zY2FsYQ==) | 0.00% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/20267?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #20267 +/- ##
============================================
- Coverage 80.62% 76.49% -4.14%
+ Complexity 35055 33331 -1724
============================================
Files 2552 2552
Lines 143400 143407 +7
Branches 17454 17455 +1
============================================
- Hits 115615 109697 -5918
- Misses 19867 26266 +6399
+ Partials 7918 7444 -474
```
| [Components](https://app.codecov.io/gh/apache/hudi/pull/20267/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/20267/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.66% <ø> (-0.50%)` | :arrow_down: |
| [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/20267/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `82.89% <ø> (-0.93%)` | :arrow_down: |
| [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/20267/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.96% <ø> (ø)` | |
| [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/20267/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `49.18% <22.22%> (-24.88%)` | :arrow_down: |
| [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/20267/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `78.19% <ø> (ø)` | |
| [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/20267/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.80% <ø> (ø)` | |
| [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/20267/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.95% <ø> (-0.17%)` | :arrow_down: |
| [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/20267/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `76.16% <ø> (-0.03%)` | :arrow_down: |
| [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/20267/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.43% <ø> (ø)` | |
| [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/20267/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.41% <ø> (ø)` | |
| [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/20267/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.00% <ø> (ø)` | |
| [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/20267/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.20% <ø> (ø)` | |
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/20267/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [flink-integration-tests](https://app.codecov.io/gh/apache/hudi/pull/20267/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `49.53% <ø> (+<0.01%)` | :arrow_up: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/20267/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.96% <ø> (-0.02%)` | :arrow_down: |
| [integration-tests](https://app.codecov.io/gh/apache/hudi/pull/20267/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `13.45% <0.00%> (-0.01%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/20267/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.69% <22.22%> (-0.01%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/20267/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `35.02% <22.22%> (-12.50%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/20267?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...a/org/apache/hudi/PartitionStatsIndexSupport.scala](https://app.codecov.io/gh/apache/hudi/pull/20267?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fhudi%2FPartitionStatsIndexSupport.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvaHVkaS9QYXJ0aXRpb25TdGF0c0luZGV4U3VwcG9ydC5zY2FsYQ==) | `72.97% <0.00%> (-1.36%)` | :arrow_down: |
| [...cala/org/apache/hudi/ColumnStatsIndexSupport.scala](https://app.codecov.io/gh/apache/hudi/pull/20267?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fhudi%2FColumnStatsIndexSupport.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvaHVkaS9Db2x1bW5TdGF0c0luZGV4U3VwcG9ydC5zY2FsYQ==) | `66.05% <25.00%> (-1.72%)` | :arrow_down: |
... and [226 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/20267/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org