You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2023/04/20 23:37:00 UTC

[jira] [Updated] (HUDI-6117) Parallelize creation of initial file groups for MDT partitions

     [ https://issues.apache.org/jira/browse/HUDI-6117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated HUDI-6117:
---------------------------------
    Labels: pull-request-available  (was: )

> Parallelize creation of initial file groups for MDT partitions
> --------------------------------------------------------------
>
>                 Key: HUDI-6117
>                 URL: https://issues.apache.org/jira/browse/HUDI-6117
>             Project: Apache Hudi
>          Issue Type: Improvement
>            Reporter: Prashant Wason
>            Assignee: Prashant Wason
>            Priority: Major
>              Labels: pull-request-available
>
> When there are large number of file groups in a MDT partition (record index having billions of records can have 10k+ partitions, creating the initial log files in a for-loop can take a long time (100ms per create = 1000 seconds = 16mins) but routinely this latency is as high as 500msec / create.
> The initial file group creation can be optimized to be done parallelly speeding up MDT partition initialization.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)