You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 11:56:20 UTC
[I] RLI & secondary index column file group counts can scale horizontally [hudi]
hudi-bot opened a new issue, #16912:
URL: https://github.com/apache/hudi/issues/16912
h3. Motivation
As of today for RLI we only set a fix number of file groups at all times. Individual file group size will keep expanding as the size of the record count increase. The size of individual file group will go unbounded which does not scale.
To address this issue, we need to figure out a way to dynamically repartition the existing file groups to a larger / smaller amount of file groups as needed. * Minimum downtime / exclusive access blocking all other operations while we scaling\
* basic correctness
* trigger of the scale up / down
* Integration with other operations on MDT
This should work for both RLI and secondary index on data columns.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-9163
- Type: Bug
- Fix version(s):
- 1.1.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] RLI & secondary index column file group counts can scale horizontally [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #16912:
URL: https://github.com/apache/hudi/issues/16912#issuecomment-5391427828
This issue was reviewed as part of the JIRA-migrated backlog triage (HUDI-9163).
**Findings: routed to the performance validation pass.**
No commit referencing HUDI-9163 exists in the tree, so nothing has landed under this ticket's key.
The claim here is quantitative, so it needs measurement on a stated dataset and hardware rather than a code reading - and any result recorded on the ticket should carry both, otherwise it will not be reproducible or comparable later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org