You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "cshuo (via GitHub)" <gi...@apache.org> on 2026/10/28 03:39:11 UTC
[PR] fix(spark): skip partition stats pruning for historical snapshots [hudi]
cshuo opened a new pull request, #20451:
URL: https://github.com/apache/hudi/pull/20451
### Describe the issue this Pull Request addresses
Closes #20443
A Spark time travel query can lose matching rows when latest-snapshot partition statistics have narrowed past a value in the requested historical snapshot. For example, after a value changes from 10 to 50, pruning a historical `value = 10` query using the latest `[50, 50]` partition bounds skips the matching partition.
### Summary and Changelog
- Skip partition-statistics pruning for historical snapshots using `PartitionStatsIndexSupport.supportsQueryType`, including the expression index partition-statistics fallback.
- Preserve partition-predicate pruning and statistics pruning for the latest snapshot.
- Add COW and MOR regression coverage for historical reads and latest-snapshot pruning.
### Impact
Time travel queries avoid incorrect partition pruning. Historical queries may scan more partitions when data skipping is enabled.
### Risk Level
Low: the change conservatively disables partition-statistics pruning for historical snapshots. The regression test reproduces the missing row on both COW and MOR before the fix and passes after it. The full `TestTimeTravelQuery` suite passed (14 tests), as did the Scala style check.
### Documentation Update
None.
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): skip partition stats pruning for time travel queries [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on PR #20451:
URL: https://github.com/apache/hudi/pull/20451#issuecomment-6470661203
> Seems reasonable for us. So, time travel queries will lose data skipping performance benefits since MDT data is built against snapshots instead of timetravel instants.
Yes, the metadata table is currently read at its latest snapshot, so its partition stats may not reflect historical data. This PR disables partition stats pruning for time travel queries targeting an earlier snapshot; partition-predicate pruning and file-level data skipping remain available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): skip partition stats pruning for time travel queries [hudi]
Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous commented on PR #20451:
URL: https://github.com/apache/hudi/pull/20451#issuecomment-6635377192
Gonna merge this in. I don't see a better alternative since our MDT is always built with the latest snapshot and there's no robust versioning in MDT. So, TLDR, MDT reads always see the latest state, and there's no as-of read today.
To achieve this feature, it's quite challenging imo, DT cleaning needs to be sync with MDT compaction/cleaning. So if cleaner config says we keep last N commits, (i.e. we can time travel to any of the N commits), MDT cleaning will also need to keep N commits worth of changes. This is not possible today.
Given there's no better alternative
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): skip partition stats pruning for time travel queries [hudi]
Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous commented on PR #20451:
URL: https://github.com/apache/hudi/pull/20451#issuecomment-6635120995
merging this in as i feel there's no better alternative unless MDT allows time travel too, which it currently does not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): skip partition stats pruning for time travel queries [hudi]
Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous merged PR #20451:
URL: https://github.com/apache/hudi/pull/20451
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): skip partition stats pruning for time travel queries [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on PR #20451:
URL: https://github.com/apache/hudi/pull/20451#issuecomment-6468617651
Seems reasonable for us. So, time travel queries will lose data skipping performance benefits since MDT data is built against snapshots instead of timetravel instants.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): skip partition stats pruning for time travel queries [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #20451:
URL: https://github.com/apache/hudi/pull/20451#issuecomment-6460364516
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/20451?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 80.79%. Comparing base ([`f39f997`](https://app.codecov.io/gh/apache/hudi/commit/f39f99713f9f8327d403947f4e38aad4259eb55a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`912600f`](https://app.codecov.io/gh/apache/hudi/commit/912600f0ececa96f0dca76387e9563758419ba43?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 3 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #20451 +/- ##
============================================
- Coverage 80.83% 80.79% -0.04%
- Complexity 35957 35971 +14
============================================
Files 2575 2575
Lines 146962 147090 +128
Branches 18032 18068 +36
============================================
+ Hits 118796 118841 +45
- Misses 20004 20074 +70
- Partials 8162 8175 +13
```
| [Components](https://app.codecov.io/gh/apache/hudi/pull/20451/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/20451/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `84.18% <ø> (ø)` | |
| [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/20451/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `84.03% <ø> (ø)` | |
| [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/20451/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `86.16% <ø> (ø)` | |
| [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/20451/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `74.20% <ø> (-0.01%)` | :arrow_down: |
| [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/20451/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `79.23% <ø> (ø)` | |
| [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/20451/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.80% <ø> (ø)` | |
| [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/20451/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.69% <ø> (-0.49%)` | :arrow_down: |
| [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/20451/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `77.31% <ø> (ø)` | |
| [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/20451/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.48% <ø> (ø)` | |
| [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/20451/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.41% <ø> (ø)` | |
| [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/20451/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.01% <ø> (ø)` | |
| [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/20451/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.90% <ø> (ø)` | |
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/20451/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/20451/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.94% <ø> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/20451?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...c/main/scala/org/apache/hudi/HoodieFileIndex.scala](https://app.codecov.io/gh/apache/hudi/pull/20451?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fhudi%2FHoodieFileIndex.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvaHVkaS9Ib29kaWVGaWxlSW5kZXguc2NhbGE=) | `81.35% <ø> (-0.19%)` | :arrow_down: |
| [...a/org/apache/hudi/PartitionStatsIndexSupport.scala](https://app.codecov.io/gh/apache/hudi/pull/20451?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fhudi%2FPartitionStatsIndexSupport.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvaHVkaS9QYXJ0aXRpb25TdGF0c0luZGV4U3VwcG9ydC5zY2FsYQ==) | `74.32% <ø> (ø)` | |
... and [2 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/20451/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): skip partition stats pruning for time travel queries [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on PR #20451:
URL: https://github.com/apache/hudi/pull/20451#issuecomment-6468633192
@nsivabalan Can you PTAL?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org