You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2022/03/21 18:50:50 UTC
[spark] branch master updated: [SPARK-38606][DOC] Update document to make a good guide of multiple versions of the Spark Shuffle Service
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new a876f00 [SPARK-38606][DOC] Update document to make a good guide of multiple versions of the Spark Shuffle Service
a876f00 is described below
commit a876f005ecef77d94bb8048fa1ff0841b7f3607a
Author: zhangxudong1 <zh...@xiaomi.com>
AuthorDate: Mon Mar 21 13:48:20 2022 -0500
[SPARK-38606][DOC] Update document to make a good guide of multiple versions of the Spark Shuffle Service
### What changes were proposed in this pull request?
Update document "Running multiple versions of the Spark Shuffle Service" to use colon when writing %s.classpath instead of commas.
### Why are the changes needed?
User may be confused when they following the current document to deploy multi-versions Spark Shuffle Service on YARN.
We have tried to run multiple versions of the Spark Shuffle Service according https://github.com/apache/spark/blob/master/docs/running-on-yarn.md
but, it wont work.
Then we solved it by using colon when writing %s.classpath instead of commas.
Related discussing is in
https://issues.apache.org/jira/browse/YARN-4577?focusedCommentId=17493624&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17493624
### Does this PR introduce _any_ user-facing change?
User document changes.
### How was this patch tested?
Closes #35914 from TonyDoen/SPARK-38606.
Authored-by: zhangxudong1 <zh...@xiaomi.com>
Signed-off-by: Sean Owen <sr...@gmail.com>
---
docs/running-on-yarn.md | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md
index 63c0376..48b0c7d 100644
--- a/docs/running-on-yarn.md
+++ b/docs/running-on-yarn.md
@@ -925,8 +925,14 @@ have configuration like:
```properties
yarn.nodemanager.aux-services = spark_shuffle_x,spark_shuffle_y
- yarn.nodemanager.aux-services.spark_shuffle_x.classpath = /path/to/spark-x-yarn-shuffle.jar,/path/to/spark-x-config
- yarn.nodemanager.aux-services.spark_shuffle_y.classpath = /path/to/spark-y-yarn-shuffle.jar,/path/to/spark-y-config
+ yarn.nodemanager.aux-services.spark_shuffle_x.classpath = /path/to/spark-x-path/fat.jar:/path/to/spark-x-config
+ yarn.nodemanager.aux-services.spark_shuffle_y.classpath = /path/to/spark-y-path/fat.jar:/path/to/spark-y-config
+```
+Or
+```properties
+ yarn.nodemanager.aux-services = spark_shuffle_x,spark_shuffle_y
+ yarn.nodemanager.aux-services.spark_shuffle_x.classpath = /path/to/spark-x-path/*:/path/to/spark-x-config
+ yarn.nodemanager.aux-services.spark_shuffle_y.classpath = /path/to/spark-y-path/*:/path/to/spark-y-config
```
The two `spark-*-config` directories each contain one file, `spark-shuffle-site.xml`. These are XML
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org