You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "Prashant Wason (Jira)" <ji...@apache.org> on 2023/04/20 23:31:00 UTC

[jira] [Created] (HUDI-6117) Parallelize creation of initial file groups for MDT partitions

Prashant Wason created HUDI-6117:
------------------------------------

             Summary: Parallelize creation of initial file groups for MDT partitions
                 Key: HUDI-6117
                 URL: https://issues.apache.org/jira/browse/HUDI-6117
             Project: Apache Hudi
          Issue Type: Improvement
            Reporter: Prashant Wason
            Assignee: Prashant Wason


When there are large number of file groups in a MDT partition (record index having billions of records can have 10k+ partitions, creating the initial log files in a for-loop can take a long time (100ms per create = 1000 seconds = 16mins) but routinely this latency is as high as 500msec / create.

The initial file group creation can be optimized to be done parallelly speeding up MDT partition initialization.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)