You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 12:46:52 UTC
[I] Use partitioned rli for pruning on the reader [hudi]
hudi-bot opened a new issue, #17108:
URL: https://github.com/apache/hudi/issues/17108
Partitioned rli is not implemented for the spark reader to prune.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-9631
- Type: New Feature
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Use partitioned rli for pruning on the reader [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #17108:
URL: https://github.com/apache/hudi/issues/17108#issuecomment-5391602442
This issue was reviewed as part of the JIRA-migrated backlog triage (HUDI-9631).
**Findings: this is implemented on `master`. The change that delivered it carried no JIRA key, which is why this ticket stayed open.**
Delivered by PR #19013, `feat(spark): Support data skipping based on partitioned RLI` (`ac179e48a0c1`, merged 2026-07-15).
The wiring is complete, not partial. `HoodieFileIndex.scala:123` makes `RecordLevelIndexSupport.create(...)` the first entry of `indicesSupport`, and that factory dispatches on whether the RLI is partitioned:
```scala
// RecordLevelIndexSupport.scala:167
def create(spark: SparkSession, metadataConfig: HoodieMetadataConfig, metaClient: HoodieTableMetaClient): RecordLevelIndexSupport = {
val isPartitioned = metaClient.getIndexForMetadataPartition(PARTITION_NAME_RECORD_INDEX)
.map[Boolean](indexDef => HoodieRecordIndex.isPartitioned(indexDef))
.orElse(false)
if (isPartitioned) {
new PartitionedRecordLevelIndexSupport(spark, metadataConfig, metaClient)
} else {
new GlobalRecordLevelIndexSupport(spark, metadataConfig, metaClient)
}
}
```
And `PartitionedRecordLevelIndexSupport.lookupCandidateFilesForRecordKeys` is a real implementation rather than a placeholder - it performs partition-scoped metadata-table lookups, applies a `MAX_PARTITIONS` threshold, and falls back with `Option.empty` in the cases where pruning cannot be determined, so it under-prunes rather than over-prunes.
Test coverage landed with it: `TestRecordLevelIndexWithSQL.scala` was added, alongside changes to `RecordLevelIndexTestBase.scala` and `TestGlobalRecordLevelIndexWithSQL.scala`, covering partitioned-RLI creation, data skipping, partition filters and the threshold fallback.
**One clarification, since it is easy to reach the wrong conclusion here.** PR #13603 is merged, but it is not the fix for this ticket - it is HUDI-9630, the writer side, and its own description is what created this one:
> HUDI-9631 ticket created to support pruning on the reader. Right now this only is implemented for the writer to use this as an index.
Separately, #17116 (HUDI-9648) remains open and is *not* a blocker for this: it asks for `readPartitionedRecordIndexLocation` APIs over new `RawKey` types that still do not exist. That is distinct work.
Closing as fixed. Worth resolving JIRA HUDI-9631 to match.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Use partitioned rli for pruning on the reader [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy closed issue #17108: Use partitioned rli for pruning on the reader
URL: https://github.com/apache/hudi/issues/17108
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org