You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "prashantwason (via GitHub)" <gi...@apache.org> on 2026/10/05 16:02:20 UTC
Re: [PR] fix(spark): avoid full partition listing in incremental reads [hudi]
prashantwason commented on PR #19898:
URL: https://github.com/apache/hudi/pull/19898#issuecomment-5998198145
Thanks @yihua for reproducing this, and sorry about the missing import — the branch was missing `import org.apache.hudi.core.read.BaseHoodieTableFileIndex`. Addressed in 4d63681:
- Fixed the build (missing import).
- Partition filters are evaluated again, against `modifiedPartitions` only: I extracted the predicate evaluation from `listMatchingPartitionPaths` into a shared helper and added `SparkHoodieTableFileIndex.filterPartitionPaths(paths, predicates)`, used by `listFileSplits` in V1 and V2. This keeps the O(modified partitions) listing without changing results.
- `filterFileSlices` override now keeps the base contract (returns the `PartitionPath`, keeps `isPartitionPruneOnly`). The index also overrides `hasPredicatesPushedDown` instead of setting the flag in the constructor, since `refresh()` resets it.
- `sizeInBytes`: handled by the companion #19912 (they should land together), noted in the description.
- Added `TestIncrementalReadWithPartitionFilter`: a range touching two partitions, filtering on each, for COW and MOR.
- Moved the description into the PR template so `validate-pr` passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org