[jira] [Created] (HUDI-9163) RLI & secondary index column file group counts can scale horizontally

["Davis Zhang (Jira)" <ji...@apache.org>]

Davis Zhang created HUDI-9163:
---------------------------------

             Summary: RLI & secondary index column file group counts can scale horizontally
                 Key: HUDI-9163
                 URL: https://issues.apache.org/jira/browse/HUDI-9163
             Project: Apache Hudi
          Issue Type: Bug
            Reporter: Davis Zhang
             Fix For: 1.1.0


h3. Motivation
As of today for RLI we only set a fix number of file groups at all times. Individual file group size will keep expanding as the size of the record count increase. The size of individual file group will go unbounded which does not scale.
 
To address this issue, we need to figure out a way to dynamically repartition the existing file groups to a larger / smaller amount of file groups as needed. * Minimum downtime / exclusive access blocking all other operations while we scaling\
 * basic correctness
 * trigger of the scale up / down
 * Integration with other operations on MDT

 
This should work for both RLI and secondary index on data columns.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)