[incubator-celeborn] branch branch-0.3 updated: [CELEBORN-789] Increase default value of flushBuffer's max components

[zh...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

zhouky pushed a commit to branch branch-0.3
in repository https://gitbox.apache.org/repos/asf/incubator-celeborn.git


The following commit(s) were added to refs/heads/branch-0.3 by this push:
     new 1be71ebbc [CELEBORN-789] Increase default value of flushBuffer's max components
1be71ebbc is described below

commit 1be71ebbc8f908bd6761f8507a75218559027c64
Author: zky.zhoukeyong <zk...@alibaba-inc.com>
AuthorDate: Wed Jul 12 20:18:48 2023 +0800

    [CELEBORN-789] Increase default value of flushBuffer's max components
    
    ### What changes were proposed in this pull request?
    Set default value of ```celeborn.worker.push.compositeBuffer.maxComponents``` to 256, to be aligned with 0.2.1-incubating version.
    
    ### Why are the changes needed?
    
    Default 16 is too small, and causes ~~severe GC~~ and CPU high load.
    
    <img width="1719" alt="image" src="https://github.com/apache/incubator-celeborn/assets/26535726/9ab9675e-c19e-44f1-af46-90c29dc4df75">
    
    ### Does this PR introduce _any_ user-facing change?
    No, it's internal config.
    
    ### How was this patch tested?
    Passes GA.
    
    Closes #1707 from waitinfuture/789.
    
    Authored-by: zky.zhoukeyong <zk...@alibaba-inc.com>
    Signed-off-by: zky.zhoukeyong <zk...@alibaba-inc.com>
    (cherry picked from commit dcf6be29d8422f3ff678639cc513d293e546593b)
    Signed-off-by: zky.zhoukeyong <zk...@alibaba-inc.com>
---
 common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala b/common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala
index c4948c41d..3be453747 100644
--- a/common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala
+++ b/common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala
@@ -2600,7 +2600,7 @@ object CelebornConf extends Logging {
         "When set to 1, Netty's direct memory is close to disk buffer, but performance " +
         "might decrease due to frequent memory copy during compaction.")
       .intConf
-      .createWithDefault(16)
+      .createWithDefault(128)
 
   val WORKER_FETCH_HEARTBEAT_ENABLED: ConfigEntry[Boolean] =
     buildConf("celeborn.worker.fetch.heartbeat.enabled")