You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@uniffle.apache.org by zh...@apache.org on 2022/10/21 03:51:49 UTC
[incubator-uniffle-website] branch master updated: Add deploy docs (#7)
This is an automated email from the ASF dual-hosted git repository.
zhifgli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-uniffle-website.git
The following commit(s) were added to refs/heads/master by this push:
new 0d2236e Add deploy docs (#7)
0d2236e is described below
commit 0d2236e47c87b72241ce9be6e733a068d5907d26
Author: frankliee <88...@qq.com>
AuthorDate: Fri Oct 21 11:51:44 2022 +0800
Add deploy docs (#7)
---
docs/{intro.md => 01-intro.md} | 0
docs/{deploy.md => 02-client_guide.md} | 0
docs/{deploy.md => 03-coordinator_guide.md} | 0
docs/04-server_guide.md | 80 +++++++++++++++++++++++++++++
docs/{metrics.md => 05-metrics.md} | 0
5 files changed, 80 insertions(+)
diff --git a/docs/intro.md b/docs/01-intro.md
similarity index 100%
rename from docs/intro.md
rename to docs/01-intro.md
diff --git a/docs/deploy.md b/docs/02-client_guide.md
similarity index 100%
copy from docs/deploy.md
copy to docs/02-client_guide.md
diff --git a/docs/deploy.md b/docs/03-coordinator_guide.md
similarity index 100%
rename from docs/deploy.md
rename to docs/03-coordinator_guide.md
diff --git a/docs/04-server_guide.md b/docs/04-server_guide.md
new file mode 100644
index 0000000..a11c339
--- /dev/null
+++ b/docs/04-server_guide.md
@@ -0,0 +1,80 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+# Uniffle Shuffle Server Guide
+
+## Deploy
+This document will introduce how to deploy Uniffle shuffle servers.
+
+### Steps
+1. unzip package to RSS_HOME
+2. update RSS_HOME/bin/rss-env.sh, eg,
+ ```
+ JAVA_HOME=<java_home>
+ HADOOP_HOME=<hadoop home>
+ XMX_SIZE="80g"
+ ```
+3. update RSS_HOME/conf/server.conf, eg,
+ ```
+ rss.rpc.server.port 19999
+ rss.jetty.http.port 19998
+ rss.rpc.executor.size 2000
+ # it should be configed the same as in coordinator
+ rss.storage.type MEMORY_LOCALFILE_HDFS
+ rss.coordinator.quorum <coordinatorIp1>:19999,<coordinatorIp2>:19999
+ # local storage path for shuffle server
+ rss.storage.basePath /data1/rssdata,/data2/rssdata....
+ # it's better to config thread num according to local disk num
+ rss.server.flush.thread.alive 5
+ rss.server.flush.threadPool.size 10
+ rss.server.buffer.capacity 40g
+ rss.server.read.buffer.capacity 20g
+ rss.server.heartbeat.timeout 60000
+ rss.server.heartbeat.interval 10000
+ rss.rpc.message.max.size 1073741824
+ rss.server.preAllocation.expired 120000
+ rss.server.commit.timeout 600000
+ rss.server.app.expired.withoutHeartbeat 120000
+ # note: the default value of rss.server.flush.cold.storage.threshold.size is 64m
+ # there will be no data written to DFS if set it as 100g even rss.storage.type=MEMORY_LOCALFILE_HDFS
+ # please set proper value if DFS is used, eg, 64m, 128m.
+ rss.server.flush.cold.storage.threshold.size 100g
+ ```
+4. start Shuffle Server
+ ```
+ bash RSS_HOME/bin/start-shuffle-server.sh
+ ```
+
+## Configuration
+|Property Name|Default|Description|
+|---|---|---|
+|rss.coordinator.quorum|-|Coordinator quorum|
+|rss.rpc.server.port|-|RPC port for Shuffle server|
+|rss.jetty.http.port|-|Http port for Shuffle server|
+|rss.server.buffer.capacity|-|Max memory of buffer manager for shuffle server|
+|rss.server.memory.shuffle.highWaterMark.percentage|75.0|Threshold of spill data to storage, percentage of rss.server.buffer.capacity|
+|rss.server.memory.shuffle.lowWaterMark.percentage|25.0|Threshold of keep data in memory, percentage of rss.server.buffer.capacity|
+|rss.server.read.buffer.capacity|-|Max size of buffer for reading data|
+|rss.server.heartbeat.interval|10000|Heartbeat interval to Coordinator (ms)|
+|rss.server.flush.threadPool.size|10|Thread pool for flush data to file|
+|rss.server.commit.timeout|600000|Timeout when commit shuffle data (ms)|
+|rss.storage.type|-|Supports MEMORY_LOCALFILE, MEMORY_HDFS, MEMORY_LOCALFILE_HDFS|
+|rss.server.flush.cold.storage.threshold.size|64M| The threshold of data size for LOACALFILE and HDFS if MEMORY_LOCALFILE_HDFS is used|
+|rss.server.tags|-|The comma-separated list of tags to indicate the shuffle server's attributes. It will be used as the assignment basis for the coordinator|
+|rss.server.single.buffer.flush.enabled|false|Whether single buffer flush when size exceeded rss.server.single.buffer.flush.threshold|
+|rss.server.single.buffer.flush.threshold|64M|The threshold of single shuffle buffer flush|
+|rss.server.disk.capacity|-1|Disk capacity that shuffle server can use. If it's negative, it will use the default disk whole space|
diff --git a/docs/metrics.md b/docs/05-metrics.md
similarity index 100%
rename from docs/metrics.md
rename to docs/05-metrics.md