You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Prashant Wason (Jira)" <ji...@apache.org> on 2023/10/05 03:19:01 UTC

[jira] [Updated] (HUDI-5261) Use proper parallelism for engine context APIs

     [ https://issues.apache.org/jira/browse/HUDI-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Prashant Wason updated HUDI-5261:
---------------------------------
    Fix Version/s: 0.14.1
                       (was: 0.14.0)

> Use proper parallelism for engine context APIs
> ----------------------------------------------
>
>                 Key: HUDI-5261
>                 URL: https://issues.apache.org/jira/browse/HUDI-5261
>             Project: Apache Hudi
>          Issue Type: Improvement
>          Components: performance
>            Reporter: Raymond Xu
>            Assignee: Jonathan Vexler
>            Priority: Critical
>              Labels: pull-request-available
>             Fix For: 0.14.1
>
>
> do a global search of these APIs
> - org.apache.hudi.common.engine.HoodieEngineContext#flatMap
> - org.apache.hudi.common.engine.HoodieEngineContext#map
> and similar ones take in parallelism.
> A lot of occurrences are using number of items as parallelism, which affect performance. Parallelism should be based on num cores available in the cluster and set by user via parallelism configs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)