You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 03:24:35 UTC
[I] Performance improvement for IncrementalRelation [hudi]
hudi-bot opened a new issue, #14482:
URL: https://github.com/apache/hudi/issues/14482
When HoodieIncrSource is used to fetch the update from another Hudi table, the IncrementalRelation will be used to read the data. But it has a performance issue because the column pruning and predicate pushdown don't happen. As the result, Hudi reads too much useless data.
By enabling the column pruning and predicate pushdown, the data to read is reduced dramatically.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-2754
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-3249
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Performance improvement for IncrementalRelation [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #14482:
URL: https://github.com/apache/hudi/issues/14482#issuecomment-4795169424
This appears to have been addressed in master by [#17514](https://github.com/apache/hudi/pull/17514) (commit `180592a0ad71`, *feat(spark): implement column pruning for incremental queries*). The PR migrates `IncrementalRelationV1` and `IncrementalRelationV2` from `TableScan` to `PrunedScan`, implements `buildScan(requiredColumns)`, and enables Parquet filter pushdown plus record-level filters — addressing both the column pruning and predicate pushdown gaps described here. Could the reporter confirm on a recent master build and close if resolved?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Performance improvement for IncrementalRelation [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #14482: Performance improvement for IncrementalRelation
URL: https://github.com/apache/hudi/issues/14482
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Performance improvement for IncrementalRelation [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on issue #14482:
URL: https://github.com/apache/hudi/issues/14482#issuecomment-3630171461
### Linked PR(s)
- https://github.com/apache/hudi/pull/3985
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org