(celeborn) branch main updated: [CELEBORN-1463] Create network memory allocator with celeborn.network.memory.allocator.numArenas

[et...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

ethanfeng pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/celeborn.git


The following commit(s) were added to refs/heads/main by this push:
     new f786390e6 [CELEBORN-1463] Create network memory allocator with celeborn.network.memory.allocator.numArenas
f786390e6 is described below

commit f786390e61d9e3f37f3313b11ff1e6c084b9df17
Author: binjie yang <ya...@shizhuang-inc.com>
AuthorDate: Mon Jun 17 11:21:07 2024 +0800

    [CELEBORN-1463] Create network memory allocator with celeborn.network.memory.allocator.numArenas
    
    ### What changes were proposed in this pull request?
    Create netty pooled network memory allocator with `celeborn.network.memory.allocator.numArenas`.
    
    ### Why are the changes needed?
    Before this, when user call to build non shared pooled memory allocator will use the number of module client threads as the
    core number.
    
    Intuitively we should respect to build memory allocator with user define `celeborn.network.memory.allocator.numArenas`.
    
    ### Does this PR introduce _any_ user-facing change?
    No
    
    ### How was this patch tested?
    None
    
    Closes #2570 from zwangsheng/CELEBORN-1463.
    
    Authored-by: binjie yang <ya...@shizhuang-inc.com>
    Signed-off-by: mingji <fe...@alibaba-inc.com>
---
 .../main/java/org/apache/celeborn/common/network/util/NettyUtils.java  | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/common/src/main/java/org/apache/celeborn/common/network/util/NettyUtils.java b/common/src/main/java/org/apache/celeborn/common/network/util/NettyUtils.java
index 4c941e864..60eb44744 100644
--- a/common/src/main/java/org/apache/celeborn/common/network/util/NettyUtils.java
+++ b/common/src/main/java/org/apache/celeborn/common/network/util/NettyUtils.java
@@ -150,7 +150,8 @@ public class NettyUtils {
           allowCache && conf.getCelebornConf().networkMemoryAllocatorAllowCache());
     }
     PooledByteBufAllocator allocator =
-        createPooledByteBufAllocator(conf.preferDirectBufs(), allowCache, conf.clientThreads());
+        createPooledByteBufAllocator(
+            conf.preferDirectBufs(), allowCache, conf.getCelebornConf().networkAllocatorArenas());
     if (source != null) {
       String poolName = "default-netty-pool";
       Map<String, String> labels = new HashMap<>();