[incubator-celeborn] branch branch-0.3 updated: [CELEBORN-897] Set celeborn.network.memory.allocator.allowCache default to false

[zh...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

zhouky pushed a commit to branch branch-0.3
in repository https://gitbox.apache.org/repos/asf/incubator-celeborn.git


The following commit(s) were added to refs/heads/branch-0.3 by this push:
     new 4233ea398 [CELEBORN-897] Set celeborn.network.memory.allocator.allowCache default to false
4233ea398 is described below

commit 4233ea398e43206a8abff45008237a9cdb8fe702
Author: zky.zhoukeyong <zk...@alibaba-inc.com>
AuthorDate: Wed Aug 16 00:26:39 2023 +0800

    [CELEBORN-897] Set celeborn.network.memory.allocator.allowCache default to false
    
    ### What changes were proposed in this pull request?
    As title
    
    ### Why are the changes needed?
    I tested 1.1T and 3.3T shuffle, as well as 3T TPCDS with thread cache on and off in the shared PooledByteBufAllocator and find no
    difference:
    | Benchmark    | Cache On | Cache Off|
    | -------- | ------- |------- |
    |1.1T Shuffle| 3.7min/1.9min   |3.7min/1.9min|
    | 3.3T Shuffle| 12min/6.7min  |12min/6.2min|
    | 3T TPCDS | 2645s |2644s|
    
    And since the configuration has a big influence to the direct memory usage, see https://github.com/apache/incubator-celeborn/pull/1716 , it's very necessary to set the default value to false.
    
    ### Does this PR introduce _any_ user-facing change?
    No.
    
    ### How was this patch tested?
    Manual test.
    
    Closes #1817 from waitinfuture/897.
    
    Authored-by: zky.zhoukeyong <zk...@alibaba-inc.com>
    Signed-off-by: zky.zhoukeyong <zk...@alibaba-inc.com>
    (cherry picked from commit 57fdbf08c293534ea8b62245d1172225f41d8be2)
    Signed-off-by: zky.zhoukeyong <zk...@alibaba-inc.com>
---
 common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala b/common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala
index 8a351d493..e08517f60 100644
--- a/common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala
+++ b/common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala
@@ -1244,7 +1244,7 @@ object CelebornConf extends Logging {
       .version("0.3.1")
       .doc("When false, globally disable thread-local cache in the shared PooledByteBufAllocator.")
       .booleanConf
-      .createWithDefault(true)
+      .createWithDefault(false)
 
   val NETWORK_MEMORY_ALLOCATOR_SHARE: ConfigEntry[Boolean] =
     buildConf("celeborn.network.memory.allocator.share")