You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@celeborn.apache.org by zh...@apache.org on 2023/04/24 13:03:57 UTC
[incubator-celeborn] branch main updated: [CELEBORN-549] Update readme about deploy flink client. (#1454)
This is an automated email from the ASF dual-hosted git repository.
zhouky pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-celeborn.git
The following commit(s) were added to refs/heads/main by this push:
new 537fc94df [CELEBORN-549] Update readme about deploy flink client. (#1454)
537fc94df is described below
commit 537fc94df298b22479159e50579228491a4b7353
Author: Ethan Feng <et...@apache.org>
AuthorDate: Mon Apr 24 21:03:53 2023 +0800
[CELEBORN-549] Update readme about deploy flink client. (#1454)
---
README.md | 84 ++++++++++++++++++++++++++++++++++++++++++++++++++++++---------
1 file changed, 72 insertions(+), 12 deletions(-)
diff --git a/README.md b/README.md
index edf5211c2..572df52e4 100644
--- a/README.md
+++ b/README.md
@@ -41,37 +41,55 @@ Celeborn Worker's slot count is decided by `total usable disk size / average shu
Celeborn worker's slot count decreases when a partition is allocated and increments when a partition is freed.
## Build
-Celeborn supports Spark 2.4/3.0/3.1/3.2/3.3 and only tested under Java 8.
+1.Celeborn supports Spark 2.4/3.0/3.1/3.2/3.3 and flink 1.14.
+2.Celeborn tested under Java 8 environment.
-Build for Spark
-```
-./build/make-distribution.sh -Pspark-2.4/-Pspark-3.0/-Pspark-3.1/-Pspark-3.2/-Pspark-3.3
+Build Celeborn
+```shell
+./build/make-distribution.sh -Pspark-2.4/-Pspark-3.0/-Pspark-3.1/-Pspark-3.2/-Pspark-3.3/-Pflink-1.14
```
package apache-celeborn-${project.version}-bin.tgz will be generated.
### Package Details
Build procedure will create a compressed package.
+
+Spark package layout:
```
├── RELEASE
├── bin
├── conf
+ ├── jars // common jars for master and worker
├── master-jars
├── worker-jars
- ├── sbin
+ ├── sbin
└── spark // Spark client jars
```
+Flink package layout:
+```
+ ├── RELEASE
+ ├── bin
+ ├── conf
+ ├── jars // common jars for master and worker
+ ├── master-jars
+ ├── worker-jars
+ ├── sbin
+ └── flink // flink client jars
+```
+
### Compatibility
-Celeborn server is compatible with all supported Spark versions.
-You can run different Spark versions with the same Celeborn server. It doesn't matter whether Celeborn server is compiled with -Pspark-2.4/3.0/3.1/3.2/3.3.
-However, Celeborn client must be consistent with the version of the Spark.
-For example, if you are running Spark 2.4, you must compile Celeborn client with -Pspark-2.4; if you are running Spark 3.2, you must compile Celeborn client with -Pspark-3.2.
+Celeborn server is compatible with all clients inside various engines.
+However, Celeborn client must be consistent with the version of the specified engine.
+For example, if you are running Spark 2.4, you must compile Celeborn client with -Pspark-2.4;
+if you are running Spark 3.2, you must compile Celeborn client with -Pspark-3.2;
+if you are running flink 1.14, you must compile Celeborn client with -Pflink-1.14.
## Usage
Celeborn cluster composes of Master and Worker nodes, the Master supports both single and HA mode(Raft-based) deployments.
### Deploy Celeborn
+#### Deploy on host
1. Unzip the tarball to `$CELEBORN_HOME`
2. Modify environment variables in `$CELEBORN_HOME/conf/celeborn-env.sh`
@@ -95,7 +113,7 @@ celeborn.master.port 9097
celeborn.metrics.enabled true
celeborn.worker.flush.buffer.size 256k
-celeborn.worker.storage.dirs /mnt/disk1/,/mnt/disk2
+celeborn.worker.storage.dirs /mnt/disk1:disktype=SSD,/mnt/disk2:disktype=HDD
# If your hosts have disk raid or use lvm, set celeborn.worker.monitor.disk.enabled to false
celeborn.worker.monitor.disk.enabled false
```
@@ -123,10 +141,24 @@ celeborn.ha.master.ratis.raft.server.storage.dir /mnt/disk1/rss_ratis/
celeborn.metrics.enabled true
# If you want to use HDFS as shuffle storage, make sure that flush buffer size is at least 4MB or larger.
celeborn.worker.flush.buffer.size 256k
-celeborn.worker.storage.dirs /mnt/disk1/,/mnt/disk2
+celeborn.worker.storage.dirs /mnt/disk1:disktype=SSD,/mnt/disk2:disktype=HDD
# If your hosts have disk raid or use lvm, set celeborn.worker.monitor.disk.enabled to false
celeborn.worker.monitor.disk.enabled false
```
+
+Flink engine related configurations:
+```properties
+# if you are using Celeborn for flink, these settings will be needed
+celeborn.worker.directMemoryRatioForReadBuffer 0.4
+celeborn.worker.directMemoryRatioToResume 0.5
+# these setting will affect performance.
+# If there is enough off-heap memory you can try to increase read buffers.
+# Read buffer max memory usage for a data partition is `taskmanager.memory.segment-size * readBuffersMax`
+celeborn.worker.partition.initial.readBuffersMin 512
+celeborn.worker.partition.initial.readBuffersMax 1024
+celeborn.worker.readBuffer.allocationWait 10ms
+```
+
4. Copy Celeborn and configurations to all nodes
5. Start all services. If you install Celeborn distribution in same path on every node and your
cluster can perform SSH login then you can fill `$CELEBORN_HOME/conf/hosts` and
@@ -156,10 +188,13 @@ Disks: {/mnt/disk1=DiskInfo(maxSlots: 6679, committed shuffles 0 shuffleAllocati
WorkerRef: null
```
+#### Deploy Celeborn on K8S
+Please refer our [website](https://celeborn.apache.org/docs/latest/deploy_on_k8s/)
+
### Deploy Spark client
Copy $CELEBORN_HOME/spark/*.jar to $SPARK_HOME/jars/
-### Spark Configuration
+#### Spark Configuration
To use Celeborn, following spark configurations should be added.
```properties
spark.shuffle.manager org.apache.spark.shuffle.celeborn.RssShuffleManager
@@ -188,6 +223,31 @@ spark.sql.adaptive.enabled true
spark.sql.adaptive.skewJoin.enabled true
```
+### Deploy Flink client
+Copy $CELEBORN_HOME/flink/*.jar to $FLINK_HOME/lib/
+
+#### Flink Configuration
+TO use Celeborn, follow flink configurations should be added.
+```properties
+shuffle-service-factory.class: org.apache.celeborn.plugin.flink.RemoteShuffleServiceFactory
+celeborn.master.endpoints: clb-1:9097,clb-2:9098,clb-3:9099
+
+celeborn.shuffle.batchHandleReleasePartition.enabled: true
+celeborn.push.maxReqsInFlight: 128
+
+# network connections between peers
+celeborn.data.io.numConnectionsPerPeer: 16
+# threads number may vary according to your cluster but do not set to 1
+celeborn.data.io.threads: 32
+celeborn.shuffle.batchHandleCommitPartition.threads: 32
+celeborn.rpc.dispatcher.numThreads: 32
+
+# floating buffers may need to change `taskmanager.network.memory.fraction` and `taskmanager.network.memory.max`
+taskmanager.network.memory.floating-buffers-per-gate: 4096
+taskmanager.network.memory.buffers-per-channel: 0
+taskmanager.memory.task.off-heap.size: 512m
+```
+
### Best Practice
If you want to set up a production-ready Celeborn cluster, your cluster should have at least 3 masters and at least 4 workers.
Masters and works can be deployed on the same node but should not deploy multiple masters or workers on the same node.