(celeborn) branch branch-0.5 updated: [CELEBORN-1403] Update celeborn configuration about offheap memory

[et...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

ethanfeng pushed a commit to branch branch-0.5
in repository https://gitbox.apache.org/repos/asf/celeborn.git


The following commit(s) were added to refs/heads/branch-0.5 by this push:
     new 4eea40e1e [CELEBORN-1403] Update celeborn configuration about offheap memory
4eea40e1e is described below

commit 4eea40e1ee82c9e9a394c40c7142ca9edf17030b
Author: mingji <fe...@alibaba-inc.com>
AuthorDate: Tue Jun 11 15:44:25 2024 +0800

    [CELEBORN-1403] Update celeborn configuration about offheap memory
    
    ### What changes were proposed in this pull request?
    Update configuration guide.
    
    ### Why are the changes needed?
    Remove outdated informations about worker's offheap memory.
    
    ### Does this PR introduce _any_ user-facing change?
    NO.
    
    ### How was this patch tested?
    No need to test.
    
    Closes #2552 from FMX/b1403.
    
    Authored-by: mingji <fe...@alibaba-inc.com>
    Signed-off-by: mingji <fe...@alibaba-inc.com>
    (cherry picked from commit 22ecbbd157c972141080f67f0bec12527c4537a0)
    Signed-off-by: mingji <fe...@alibaba-inc.com>
---
 docs/configuration/index.md | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/docs/configuration/index.md b/docs/configuration/index.md
index 9d620a0b5..c657f32b5 100644
--- a/docs/configuration/index.md
+++ b/docs/configuration/index.md
@@ -36,14 +36,17 @@ Off-heap memory requirement can be estimated as below:
 ```
 numDirs = `celeborn.worker.storage.dirs`             # the amount of directory will be used by Celeborn storage
 bufferSize = `celeborn.worker.flusher.buffer.size`   # the amount of memory will be used by a single flush buffer 
-off-heap-memory = bufferSize * estimatedTasks * 2 + network memory
+off-heap-memory = (disk buffer * disks) + network memory       # the disk buffer is a logical memory region that stores shuffle data received from network 
+                                                               # shuffle data will be flushed to disks through write tasks
+                                                               # the amount of disk buffer can be set to 1GB or larger for each disk according to the difference of your disk speed and network speed
 ```
 
-For example, if a Celeborn worker has 10 storage directories or disks and the buffer size is set to 256 KiB.
-The necessary off-heap memory is 10 GiB.
+For example, if a Celeborn worker give each disk 1GiB memory and the buffer size is set to 256 KB. 
+Celeborn worker can support up to 4096 concurrent write tasks for each disk.  
+If this worker has 10 disks, the offheap memory should be set to 12GB.
 
 Network memory will be consumed when netty reads from a TCP channel, there will need some extra
-memory. Empirically, Celeborn worker off-heap memory should be set to `(numDirs  * bufferSize * 1.2)`.
+memory. Empirically, Celeborn worker off-heap memory should be set to `((disk buffer * disks) * 1.2)`.
 
 ## All Configurations