You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 05:56:37 UTC
[I] Fix clean planning for very large partitions [hudi]
hudi-bot opened a new issue, #15479:
URL: https://github.com/apache/hudi/issues/15479
Within clean planning phase, we do a map() for every partition and then trigger planning for each partition within that.
For very large number of partitions, and if cleaner shuffle parallelism is small, this results in more sequential planning. We can enhance this with mapPartitions call and optimize it
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5012
- Type: Improvement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fix clean planning for very large partitions [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #15479:
URL: https://github.com/apache/hudi/issues/15479#issuecomment-5056513619
This improvement was reviewed as part of the JIRA-migrated backlog triage.
The ask - use a mapPartitions-style plan instead of a per-partition map() during clean planning for tables with very large partition counts - is still valid on current master. CleanPlanActionExecutor still drives planning through context.map(...) bounded by cleanerParallelism (it now batches partitions in chunks to bound timeline-server memory, but does not use a mapPartitions plan). The only linked PR, #6890, was closed unmerged, so this was never implemented.
Fix scope is larger than a trivial patch and needs rebase/rework. Keeping open and tracking as stream:fix / clean_archive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fix clean planning for very large partitions [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on issue #15479:
URL: https://github.com/apache/hudi/issues/15479#issuecomment-3630149018
### Linked PR(s)
- https://github.com/apache/hudi/pull/6890
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org