You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@celeborn.apache.org by zh...@apache.org on 2023/04/24 13:03:57 UTC

[incubator-celeborn] branch main updated: [CELEBORN-549] Update readme about deploy flink client. (#1454)

This is an automated email from the ASF dual-hosted git repository.

zhouky pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-celeborn.git


The following commit(s) were added to refs/heads/main by this push:
     new 537fc94df [CELEBORN-549] Update readme about deploy flink client. (#1454)
537fc94df is described below

commit 537fc94df298b22479159e50579228491a4b7353
Author: Ethan Feng <et...@apache.org>
AuthorDate: Mon Apr 24 21:03:53 2023 +0800

    [CELEBORN-549] Update readme about deploy flink client. (#1454)
---
 README.md | 84 ++++++++++++++++++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 72 insertions(+), 12 deletions(-)

diff --git a/README.md b/README.md
index edf5211c2..572df52e4 100644
--- a/README.md
+++ b/README.md
@@ -41,37 +41,55 @@ Celeborn Worker's slot count is decided by `total usable disk size / average shu
 Celeborn worker's slot count decreases when a partition is allocated and increments when a partition is freed.
 
 ## Build
-Celeborn supports Spark 2.4/3.0/3.1/3.2/3.3 and only tested under Java 8.
+1.Celeborn supports Spark 2.4/3.0/3.1/3.2/3.3 and flink 1.14.
+2.Celeborn tested under Java 8 environment.
 
-Build for Spark
-```
-./build/make-distribution.sh -Pspark-2.4/-Pspark-3.0/-Pspark-3.1/-Pspark-3.2/-Pspark-3.3
+Build Celeborn
+```shell
+./build/make-distribution.sh -Pspark-2.4/-Pspark-3.0/-Pspark-3.1/-Pspark-3.2/-Pspark-3.3/-Pflink-1.14
 ```
 
 package apache-celeborn-${project.version}-bin.tgz will be generated.
 
 ### Package Details
 Build procedure will create a compressed package.
+
+Spark package layout:
 ```
     ├── RELEASE                         
     ├── bin                             
     ├── conf                            
+    ├── jars           // common jars for master and worker                 
     ├── master-jars                     
     ├── worker-jars                     
-    ├── sbin                            
+    ├── sbin
     └── spark          // Spark client jars
 ```
 
+Flink package layout:
+```
+    ├── RELEASE                         
+    ├── bin                             
+    ├── conf                            
+    ├── jars           // common jars for master and worker                        
+    ├── master-jars                     
+    ├── worker-jars                     
+    ├── sbin
+    └── flink          // flink client jars
+```
+
 ### Compatibility
-Celeborn server is compatible with all supported Spark versions.
-You can run different Spark versions with the same Celeborn server. It doesn't matter whether Celeborn server is compiled with -Pspark-2.4/3.0/3.1/3.2/3.3.
-However, Celeborn client must be consistent with the version of the Spark.
-For example, if you are running Spark 2.4, you must compile Celeborn client with -Pspark-2.4; if you are running Spark 3.2, you must compile Celeborn client with -Pspark-3.2.
+Celeborn server is compatible with all clients inside various engines.
+However, Celeborn client must be consistent with the version of the specified engine.
+For example, if you are running Spark 2.4, you must compile Celeborn client with -Pspark-2.4;
+if you are running Spark 3.2, you must compile Celeborn client with -Pspark-3.2; 
+if you are running flink 1.14, you must compile Celeborn client with -Pflink-1.14.
 
 ## Usage
 Celeborn cluster composes of Master and Worker nodes, the Master supports both single and HA mode(Raft-based) deployments.
 
 ### Deploy Celeborn
+#### Deploy on host
 1. Unzip the tarball to `$CELEBORN_HOME`
 2. Modify environment variables in `$CELEBORN_HOME/conf/celeborn-env.sh`
 
@@ -95,7 +113,7 @@ celeborn.master.port 9097
 
 celeborn.metrics.enabled true
 celeborn.worker.flush.buffer.size 256k
-celeborn.worker.storage.dirs /mnt/disk1/,/mnt/disk2
+celeborn.worker.storage.dirs /mnt/disk1:disktype=SSD,/mnt/disk2:disktype=HDD
 # If your hosts have disk raid or use lvm, set celeborn.worker.monitor.disk.enabled to false
 celeborn.worker.monitor.disk.enabled false
 ```   
@@ -123,10 +141,24 @@ celeborn.ha.master.ratis.raft.server.storage.dir /mnt/disk1/rss_ratis/
 celeborn.metrics.enabled true
 # If you want to use HDFS as shuffle storage, make sure that flush buffer size is at least 4MB or larger.
 celeborn.worker.flush.buffer.size 256k
-celeborn.worker.storage.dirs /mnt/disk1/,/mnt/disk2
+celeborn.worker.storage.dirs /mnt/disk1:disktype=SSD,/mnt/disk2:disktype=HDD
 # If your hosts have disk raid or use lvm, set celeborn.worker.monitor.disk.enabled to false
 celeborn.worker.monitor.disk.enabled false
 ```
+
+Flink engine related configurations:
+```properties
+# if you are using Celeborn for flink, these settings will be needed
+celeborn.worker.directMemoryRatioForReadBuffer 0.4
+celeborn.worker.directMemoryRatioToResume 0.5
+# these setting will affect performance. 
+# If there is enough off-heap memory you can try to increase read buffers.
+# Read buffer max memory usage for a data partition is `taskmanager.memory.segment-size * readBuffersMax`
+celeborn.worker.partition.initial.readBuffersMin 512
+celeborn.worker.partition.initial.readBuffersMax 1024
+celeborn.worker.readBuffer.allocationWait 10ms
+```
+
 4. Copy Celeborn and configurations to all nodes
 5. Start all services. If you install Celeborn distribution in same path on every node and your
    cluster can perform SSH login then you can fill `$CELEBORN_HOME/conf/hosts` and
@@ -156,10 +188,13 @@ Disks: {/mnt/disk1=DiskInfo(maxSlots: 6679, committed shuffles 0 shuffleAllocati
 WorkerRef: null
 ```
 
+#### Deploy Celeborn on K8S
+Please refer our [website](https://celeborn.apache.org/docs/latest/deploy_on_k8s/)
+
 ### Deploy Spark client
 Copy $CELEBORN_HOME/spark/*.jar to $SPARK_HOME/jars/
 
-### Spark Configuration
+#### Spark Configuration
 To use Celeborn, following spark configurations should be added.
 ```properties
 spark.shuffle.manager org.apache.spark.shuffle.celeborn.RssShuffleManager
@@ -188,6 +223,31 @@ spark.sql.adaptive.enabled true
 spark.sql.adaptive.skewJoin.enabled true
 ```
 
+### Deploy Flink client
+Copy $CELEBORN_HOME/flink/*.jar to $FLINK_HOME/lib/
+
+#### Flink Configuration
+TO use Celeborn, follow flink configurations should be added.
+```properties
+shuffle-service-factory.class: org.apache.celeborn.plugin.flink.RemoteShuffleServiceFactory
+celeborn.master.endpoints: clb-1:9097,clb-2:9098,clb-3:9099
+
+celeborn.shuffle.batchHandleReleasePartition.enabled: true
+celeborn.push.maxReqsInFlight: 128
+
+# network connections between peers
+celeborn.data.io.numConnectionsPerPeer: 16
+# threads number may vary according to your cluster but do not set to 1
+celeborn.data.io.threads: 32
+celeborn.shuffle.batchHandleCommitPartition.threads: 32
+celeborn.rpc.dispatcher.numThreads: 32
+
+# floating buffers may need to change `taskmanager.network.memory.fraction` and `taskmanager.network.memory.max`
+taskmanager.network.memory.floating-buffers-per-gate: 4096
+taskmanager.network.memory.buffers-per-channel: 0
+taskmanager.memory.task.off-heap.size: 512m
+```
+
 ### Best Practice
 If you want to set up a production-ready Celeborn cluster, your cluster should have at least 3 masters and at least 4 workers.
 Masters and works can be deployed on the same node but should not deploy multiple masters or workers on the same node.