You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2022/03/21 18:53:49 UTC
[spark] branch branch-3.2 updated: [SPARK-38606][DOC] Update document to make a good guide of multiple versions of the Spark Shuffle Service
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new e78038b [SPARK-38606][DOC] Update document to make a good guide of multiple versions of the Spark Shuffle Service
e78038b is described below
commit e78038b1b1b9f8de75c6cf5df6613cb5fe8ac9cf
Author: zhangxudong1 <zh...@xiaomi.com>
AuthorDate: Mon Mar 21 13:48:20 2022 -0500
[SPARK-38606][DOC] Update document to make a good guide of multiple versions of the Spark Shuffle Service
### What changes were proposed in this pull request?
Update document "Running multiple versions of the Spark Shuffle Service" to use colon when writing %s.classpath instead of commas.
### Why are the changes needed?
User may be confused when they following the current document to deploy multi-versions Spark Shuffle Service on YARN.
We have tried to run multiple versions of the Spark Shuffle Service according https://github.com/apache/spark/blob/master/docs/running-on-yarn.md
but, it wont work.
Then we solved it by using colon when writing %s.classpath instead of commas.
Related discussing is in
https://issues.apache.org/jira/browse/YARN-4577?focusedCommentId=17493624&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17493624
### Does this PR introduce _any_ user-facing change?
User document changes.
### How was this patch tested?
Closes #35914 from TonyDoen/SPARK-38606.
Authored-by: zhangxudong1 <zh...@xiaomi.com>
Signed-off-by: Sean Owen <sr...@gmail.com>
(cherry picked from commit a876f005ecef77d94bb8048fa1ff0841b7f3607a)
Signed-off-by: Sean Owen <sr...@gmail.com>
---
docs/running-on-yarn.md | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md
index 4a1dddf..77d0ed0 100644
--- a/docs/running-on-yarn.md
+++ b/docs/running-on-yarn.md
@@ -888,8 +888,14 @@ above. For example, you may have configuration like:
```properties
yarn.nodemanager.aux-services = spark_shuffle_x,spark_shuffle_y
- yarn.nodemanager.aux-services.spark_shuffle_x.classpath = /path/to/spark-x-yarn-shuffle.jar,/path/to/spark-x-config
- yarn.nodemanager.aux-services.spark_shuffle_y.classpath = /path/to/spark-y-yarn-shuffle.jar,/path/to/spark-y-config
+ yarn.nodemanager.aux-services.spark_shuffle_x.classpath = /path/to/spark-x-path/fat.jar:/path/to/spark-x-config
+ yarn.nodemanager.aux-services.spark_shuffle_y.classpath = /path/to/spark-y-path/fat.jar:/path/to/spark-y-config
+```
+Or
+```properties
+ yarn.nodemanager.aux-services = spark_shuffle_x,spark_shuffle_y
+ yarn.nodemanager.aux-services.spark_shuffle_x.classpath = /path/to/spark-x-path/*:/path/to/spark-x-config
+ yarn.nodemanager.aux-services.spark_shuffle_y.classpath = /path/to/spark-y-path/*:/path/to/spark-y-config
```
The two `spark-*-config` directories each contain one file, `spark-shuffle-site.xml`. These are XML
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org