You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@uniffle.apache.org by ro...@apache.org on 2023/07/03 07:51:00 UTC
[incubator-uniffle] branch master updated: [MINOR] docs: update document for tez client plugin. (#987)
This is an automated email from the ASF dual-hosted git repository.
roryqi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-uniffle.git
The following commit(s) were added to refs/heads/master by this push:
new ff2128f8 [MINOR] docs: update document for tez client plugin. (#987)
ff2128f8 is described below
commit ff2128f82ee58a62f2da46d0bca0dd4cbf0ea38f
Author: zhengchenyu <zh...@163.com>
AuthorDate: Mon Jul 3 15:50:55 2023 +0800
[MINOR] docs: update document for tez client plugin. (#987)
Co-authored-by: zhengchenyu001 <zh...@ke.com>
---
README.md | 20 +++++++++++++++++++-
docs/client_guide.md | 16 ++++++++++++++++
2 files changed, 35 insertions(+), 1 deletion(-)
diff --git a/README.md b/README.md
index a4246fb5..1f1cd078 100644
--- a/README.md
+++ b/README.md
@@ -22,10 +22,11 @@ It provides the ability to push shuffle data into centralized storage service,
changing the shuffle style from "local file pull-like style" to "remote block push-like style".
It brings in several advantages like supporting disaggregated storage deployment,
super large shuffle jobs, and high elasticity.
-Currently it supports [Apache Spark][1] and [Apache Hadoop MapReduce][2].
+Currently it supports [Apache Spark][1] and [Apache Hadoop MapReduce][2] and [Apache Tez][3].
[1]: https://spark.apache.org
[2]: https://hadoop.apache.org
+[3]: https://tez.apache.org
[](https://github.com/apache/incubator-uniffle/actions/workflows/build.yml)
[](https://codecov.io/gh/apache/incubator-uniffle)
@@ -248,6 +249,23 @@ The jar for MapReduce is located in <RSS_HOME>/jars/client/mr/rss-client-mr-XXXX
Note that the RssMRAppMaster will automatically disable slow start (i.e., `mapreduce.job.reduce.slowstart.completedmaps=1`)
and job recovery (i.e., `yarn.app.mapreduce.am.job.recovery.enable=false`)
+### Deploy Tez Client
+
+1. Append client jar to pacakge which is set by 'tez.lib.uris'.
+
+In production mode, you can append client jar (rss-client-tez-XXXXX-shaded.jar) to package which is set by 'tez.lib.uris'.
+
+In development mode, you can append client jar (rss-client-tez-XXXXX-shaded.jar) to HADOOP_CLASSPATH.
+
+2. Update tez-site.xml to enable Uniffle.
+
+| Property Name |Default| Description |
+|----------------------------|---|------------------------------|
+| tez.am.launch.cmd-opts |-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC org.apache.tez.dag.app.RssDAGAppMaster| enable remote shuffle service |
+| tez.rss.coordinator.quorum |coordinatorIp1:19999,coordinatorIp2:19999|coordinator address|
+
+Note that the RssDAGAppMaster will automatically disable slow start (i.e., `tez.shuffle-vertex-manager.min-src-fraction=1`, `tez.shuffle-vertex-manager.max-src-fraction=1`).
+
### Deploy In Kubernetes
We have provided an operator for deploying uniffle in kubernetes environments.
diff --git a/docs/client_guide.md b/docs/client_guide.md
index 7864a2cb..e34e651b 100644
--- a/docs/client_guide.md
+++ b/docs/client_guide.md
@@ -97,6 +97,22 @@ The jar for MapReduce is located in <RSS_HOME>/jars/client/mr/rss-client-mr-XXXX
Note that the RssMRAppMaster will automatically disable slow start (i.e., `mapreduce.job.reduce.slowstart.completedmaps=1`)
and job recovery (i.e., `yarn.app.mapreduce.am.job.recovery.enable=false`)
+### Deploy Tez Client Plugin
+
+1. Append client jar to pacakge which is set by 'tez.lib.uris'.
+
+In production mode, you can append client jar (rss-client-tez-XXXXX-shaded.jar) to package which is set by 'tez.lib.uris'.
+
+In development mode, you can append client jar (rss-client-tez-XXXXX-shaded.jar) to HADOOP_CLASSPATH.
+
+2. Update tez-site.xml to enable Uniffle.
+
+| Property Name |Default| Description |
+|----------------------------|---|------------------------------|
+| tez.am.launch.cmd-opts |-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC org.apache.tez.dag.app.RssDAGAppMaster| enable remote shuffle service |
+| tez.rss.coordinator.quorum |coordinatorIp1:19999,coordinatorIp2:19999|coordinator address|
+
+Note that the RssDAGAppMaster will automatically disable slow start (i.e., `tez.shuffle-vertex-manager.min-src-fraction=1`, `tez.shuffle-vertex-manager.max-src-fraction=1`).
## Configuration