You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "Raymond Xu (Jira)" <ji...@apache.org> on 2022/11/22 11:20:00 UTC

[jira] [Created] (HUDI-5261) Use proper parallelism for engine context APIs

Raymond Xu created HUDI-5261:
--------------------------------

             Summary: Use proper parallelism for engine context APIs
                 Key: HUDI-5261
                 URL: https://issues.apache.org/jira/browse/HUDI-5261
             Project: Apache Hudi
          Issue Type: Improvement
          Components: performance
            Reporter: Raymond Xu
             Fix For: 0.12.2


do a global search of these APIs
- org.apache.hudi.common.engine.HoodieEngineContext#flatMap
- org.apache.hudi.common.engine.HoodieEngineContext#map

and similar ones take in parallelism.

A lot of occurrences are using number of items as parallelism, which affect performance. Parallelism should be based on num cores available in the cluster and set by user via parallelism configs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)