You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/03/22 23:04:14 UTC
[GitHub] [spark] xkrogen opened a new pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
xkrogen opened a new pull request #31936:
URL: https://github.com/apache/spark/pull/31936
### What changes were proposed in this pull request?
Add a new config, `spark.shuffle.service.name`, which allows for Spark applications to look for a YARN shuffle service which is defined at a name other than the default `spark_shuffle`.
Add a new config, `spark.yarn.shuffle.service.metrics.namespace`, which allows for configuring the namespace used when emitting metrics from the shuffle service into the NodeManager's `metrics2` system.
Add a new mechanism by which to override shuffle service configurations independently of the configurations in the NodeManager. When a resource `spark-shuffle-site.xml` is present on the classpath of the shuffle service, the configs present within it will be used to override the configs coming from `yarn-site.xml` (via the NodeManager).
### Why are the changes needed?
There are two use cases which can benefit from these changes.
One use case is to run multiple instances of the shuffle service side-by-side in the same NodeManager. This can be helpful, for example, when running a YARN cluster with a mixed workload of applications running multiple Spark versions, since a given version of the shuffle service is not always compatible with other versions of Spark (e.g. see SPARK-27780). With this PR, it is possible to run two shuffle services like `spark_shuffle` and `spark_shuffle_3.2.0`, one of which is "legacy" and one of which is for new applications. This is possible because YARN versions since 2.9.0 support the ability to run shuffle services within an isolated classloader (see YARN-4577), meaning multiple Spark versions can coexist.
Besides this, the separation of shuffle service configs into `spark-shuffle-site.xml` can be useful for administrators who want to change and/or deploy Spark shuffle service configurations independently of the configurations for the NodeManager (e.g., perhaps they are owned by two different teams).
### Does this PR introduce _any_ user-facing change?
Yes. There are two new configurations related to the external shuffle service, and a new mechanism which can optionally be used to configure the shuffle service. `docs/running-on-yarn.md` has been updated to provide user instructions; please see this guide for more details.
### How was this patch tested?
In addition to the new unit tests added, I have deployed this to a live YARN cluster and successfully deployed two Spark shuffle services simultaneously, one running a modified version of Spark 2.3.0 (which supports some of the newer shuffle protocols) and one running Spark 3.1.1. Spark applications of both versions are able to communicate with their respective shuffle services without issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #31936:
URL: https://github.com/apache/spark/pull/31936#discussion_r600477876
##########
File path: common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
##########
@@ -139,6 +164,13 @@
private DB db;
public YarnShuffleService() {
+ // The name of the auxiliary service configured within the NodeManager
+ // (`yarn.nodemanager.aux-services`) is treated as the source-of-truth, so this one can be
Review comment:
for yarn 2.9+
##########
File path: docs/running-on-yarn.md
##########
@@ -811,3 +831,52 @@ do the following:
to the list of filters in the <code>spark.ui.filters</code> configuration.
Be aware that the history server information may not be up-to-date with the application's state.
+
+# Running multiple versions of the Spark Shuffle Service
+
+In some cases it may be desirable to run multiple instances of the Spark Shuffle Service which are
+using different versions of Spark. This can be helpful, for example, when running a YARN cluster
+with a mixed workload of applications running multiple Spark versions, since a given version of
+the shuffle service is not always compatible with other versions of Spark. YARN versions since 2.9.0
+support the ability to run shuffle services within an isolated classloader
Review comment:
I think we should be more explicit here and say requires Yarn 2.9+
##########
File path: docs/running-on-yarn.md
##########
@@ -761,8 +761,27 @@ The following extra configuration options are available when the shuffle service
NodeManagers where the Spark Shuffle Service is not running.
</td>
</tr>
+<tr>
+ <td><code>spark.yarn.shuffle.service.metrics.namespace</code></td>
+ <td><code>sparkShuffleService</code></td>
+ <td>
+ The namespace to use when emitting shuffle service metrics into Hadoop metrics2 system of the
+ NodeManager.
Review comment:
it looks like the name referenced by the node manager works with the Hadoop 2.9+ custom class loader, but I assume with Hadoop 2.7 it requires the spark_shuffle name ? hence the spark.shuffle.service.name won't work unless you have recompiled the code and manually changed it.
Perhaps we just need to be more explicit in the config spark.shuffle.service.name that either references the section running multiple versions of the Spark Shuffle Service or explicitly states supported in YARN 2.9+. I assume this config with metrics doesn't matter as far as Hadoop version.
Also did we explicitly test with Hadoop 2.7 and the case @dongjoon-hyun brings up?
##########
File path: common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
##########
@@ -75,6 +76,15 @@
* is because an application running on the same Yarn cluster may choose to not use the external
* shuffle service, in which case its setting of `spark.authenticate` should be independent of
* the service's.
+ *
+ * The shuffle service will produce metrics via the YARN NodeManager's {@code metrics2} system
+ * under a namespace specified by the {@value SPARK_SHUFFLE_SERVICE_METRICS_NAMESPACE_KEY} config.
+ *
+ * By default, all configurations for the shuffle service will be taken directly from the
+ * Hadoop {@link Configuration} passed by the YARN NodeManager. It is also possible to configure
+ * the shuffle service by placing a resource named
+ * {@value SHUFFLE_SERVICE_CONF_OVERLAY_RESOURCE_NAME} into the classpath, which should be an
Review comment:
again add comment about with YARN 2.9+
##########
File path: resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnShuffleAlternateNameConfigSuite.scala
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn
+
+import java.net.URLClassLoader
+
+import org.apache.hadoop.yarn.conf.YarnConfiguration
+
+import org.apache.spark._
+import org.apache.spark.internal.config._
+import org.apache.spark.network.yarn.{YarnShuffleService, YarnTestAccessor}
+import org.apache.spark.tags.ExtendedYarnTest
+
+/**
+ * SPARK-34828: Integration test for the external shuffle service with an alternate name and
+ * configs (by using a configuration overlay)
+ */
+@ExtendedYarnTest
+class YarnShuffleAlternateNameConfigSuite extends YarnShuffleIntegrationSuite {
+
+ private[this] val shuffleServiceName = "custom_shuffle_service_name"
+
+ override def newYarnConfig(): YarnConfiguration = {
+ val yarnConfig = super.newYarnConfig()
+ yarnConfig.set(YarnConfiguration.NM_AUX_SERVICES, shuffleServiceName)
+ yarnConfig.set(YarnConfiguration.NM_AUX_SERVICE_FMT.format(shuffleServiceName),
+ classOf[YarnShuffleService].getCanonicalName)
+ val overlayConf = new YarnConfiguration()
+ // Enable authentication in the base NodeManager conf but not in the client. This would break
+ // shuffle, unless the shuffle service conf overlay overrides to turn off authentication.
+ overlayConf.setBoolean(NETWORK_AUTH_ENABLED.key, true)
+ // Add the authentication conf to a separate config object used as an overlay rather than
+ // setting it directly. This is necessary because a config overlay will override previous
+ // config overlays, but not configs which were set directly on the config object.
+ yarnConfig.addResource(overlayConf)
+ yarnConfig
+ }
+
+ override protected def extraSparkConf(): Map[String, String] =
+ super.extraSparkConf() ++ Map(SHUFFLE_SERVICE_NAME.key -> shuffleServiceName)
Review comment:
how does this work with Hadoop 2.7 tests? am I mistaken on how 2.7 is using the name?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #31936:
URL: https://github.com/apache/spark/pull/31936#discussion_r599694896
##########
File path: docs/running-on-yarn.md
##########
@@ -761,8 +761,27 @@ The following extra configuration options are available when the shuffle service
NodeManagers where the Spark Shuffle Service is not running.
</td>
</tr>
+<tr>
+ <td><code>spark.yarn.shuffle.service.metrics.namespace</code></td>
+ <td><code>sparkShuffleService</code></td>
+ <td>
+ The namespace to use when emitting shuffle service metrics into Hadoop metrics2 system of the
+ NodeManager.
Review comment:
For this section, everything will work as expected on Hadoop 2.7.x. The "Running multiple versions" section won't work on 2.7, but I already called out the supported YARN versions there. Can you let me know if there's anything else you think I should call out?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-806010520
**[Test build #136477 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136477/testReport)** for PR 31936 at commit [`0427a2f`](https://github.com/apache/spark/commit/0427a2f7d34f8b24678feb3d8857c95ddcfda93d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-805023081
Working on addressing test failures, will update the PR soon ...
@tgravescs thanks for the insightful comments!
> this was tried once before under #26000 so just putting it here for reference. I think the main thing is if we are going to support that we make sure its truly configurable and not just half done. But it sounds like you are trying to address the concerns brought up there but I need to do a more detailed review.
Thanks a lot for sharing this, I did not find it in my previous search. I agree that the concerns raised there around supporting different configurations should be addressed by this PR. Just making the shuffle service name configurable was the easy part :)
> One thing doing a quick skim I don't see if in YarnShuffleService.java its still registering the AuxiliaryService with the name spark_shuffle, Is that not really used by the yarn auxiliary service and it uses what you configure in the node manager? In the very least that needs to be documented in the code.
Great callout. I mentioned this in the JIRA description but it looks like I forgot to include it in the PR description. The name in the NodeManager config (`yarn.nodemanager.aux-services`) is the source-of-truth for the name. The name provided by the service (hard-coded to `spark_shuffle` in our case) is compared to the configured one, and the NodeManager will log a warning if it doesn't match because it indicates you might have a misconfiguration, but then it is never used again. Actually when using the isolated classloader, the hard-coded name is completely ignored, and instead the name of the class is used:
https://github.com/apache/hadoop/blob/a89ca56a1b0eb949f56e7c6c5c25fdf87914a02f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/AuxiliaryServiceWithCustomClassLoader.java#L170-L172
I would use the value of `spark.shuffle.service.name` from the configuration instead of hard-coding it, just to keep things clean, but we need the name at the time of class instantiation, and we don't get a `Configuration` object until `serviceInit` is called, at which point it's too late. I will update the code comments accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-804508159
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40950/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-810341471
merged to master, thanks @xkrogen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] asfgit closed pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #31936:
URL: https://github.com/apache/spark/pull/31936
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-804454370
**[Test build #136366 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136366/testReport)** for PR 31936 at commit [`37b9d14`](https://github.com/apache/spark/commit/37b9d147f28668f13e405aca952327598b52eb22).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-809708363
The GitHub Actions checks have been queued all day with no progress -- so can't confirm if those are working -- but from the [SparkPullRequestBuilder output](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136669/testReport/org.apache.spark.sql/), the TPCDS tests are succeeding:
```
Class | Duration | Fail | (diff) | Skip | (diff) | Pass | (diff) | Total | (diff)
...
TPCDSV1_4_PlanStabilitySuite | 47 sec | 0 | | 0 | | 103 | | 103 |
TPCDSV1_4_PlanStabilityWithStatsSuite | 38 sec | 0 | | 0 | | 103 | | 103
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #31936:
URL: https://github.com/apache/spark/pull/31936#discussion_r599272016
##########
File path: resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnShuffleIntegrationSuite.scala
##########
@@ -109,6 +110,59 @@ class YarnShuffleAuthSuite extends YarnShuffleIntegrationSuite {
}
+/**
+ * SPARK-34828: Integration test for the external shuffle service with an alternate name and
+ * configs (by using a configuration overlay)
+ */
+@ExtendedYarnTest
+class YarnShuffleAlternateNameConfigSuite extends YarnShuffleIntegrationSuite {
Review comment:
Please make this new test suite as a separate file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen edited a comment on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
xkrogen edited a comment on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-807168081
I went through the logs for the [most recent test failure](https://github.com/apache/spark/pull/31936/checks?check_run_id=2193694302) and couldn't find any test failures:
```
> grep -E -e "completed.+aborted" -e "Passed.+Failed" 6_Build\ modules\ sql\ -\ slow\ tests\ \(JDK\ 8,\ hadoop3.2,\ hive2.3\).txt
2021-03-25T15:45:53.5726513Z [info] Suites: completed 423, aborted 0
2021-03-25T15:45:53.5834071Z [info] Passed: Total 1062, Failed 0, Errors 0, Passed 1062, Ignored 1
```
(I had to download the logs since they were too big for the GitHub web viewer).
Not sure what's wrong but this same suite passed on my previous build with only documentation changes since then (the previous failure was in a different module).
> Just to clarify did you test this on both a Hadoop < 2.9 and > 2.9 version? It sounds like you probably did but double checking.
@tgravescs -- I think I did test on a Hadoop 2.7 cluster previously, but just to be sure, I tried just now. Works great using a custom shuffle service name and conf overlay via `spark-shuffle-site.xml`. And I have tested on a Hadoop 2.10 cluster which is where I was doing the multiple-ESS testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-806265869
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41061/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #31936:
URL: https://github.com/apache/spark/pull/31936#discussion_r599687546
##########
File path: resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnShuffleIntegrationSuite.scala
##########
@@ -109,6 +110,59 @@ class YarnShuffleAuthSuite extends YarnShuffleIntegrationSuite {
}
+/**
+ * SPARK-34828: Integration test for the external shuffle service with an alternate name and
+ * configs (by using a configuration overlay)
+ */
+@ExtendedYarnTest
+class YarnShuffleAlternateNameConfigSuite extends YarnShuffleIntegrationSuite {
Review comment:
Sure. I was following the example of `YarnShuffleAuthSuite`. Moved out to a separate file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #31936:
URL: https://github.com/apache/spark/pull/31936#discussion_r600686056
##########
File path: common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
##########
@@ -139,6 +164,13 @@
private DB db;
public YarnShuffleService() {
+ // The name of the auxiliary service configured within the NodeManager
+ // (`yarn.nodemanager.aux-services`) is treated as the source-of-truth, so this one can be
Review comment:
No, all versions of YARN, as discussed above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-809402040
I'm not sure why they are failing but I think its these:
```
2021-03-26T17:53:41.9387410Z [0m[[0m[0minfo[0m] [0m[0m[31m*** 2 TESTS FAILED ***[0m[0m
2021-03-26T17:53:41.9700291Z [0m[[0m[31merror[0m] [0m[0mFailed: Total 8676, Failed 2, Errors 0, Passed 8674, Ignored 34[0m
2021-03-26T17:53:42.0008332Z [0m[[0m[31merror[0m] [0m[0mFailed tests:[0m
2021-03-26T17:53:42.0010101Z [0m[[0m[31merror[0m] [0m[0m org.apache.spark.sql.TPCDSV1_4_PlanStabilitySuite[0m
2021-03-26T17:53:42.0011919Z [0m[[0m[31merror[0m] [0m[0m org.apache.spark.sql.TPCDSV1_4_PlanStabilityWithStatsSuite[0m
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-809724460
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41251/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-809730491
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41251/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-809740666
**[Test build #136670 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136670/testReport)** for PR 31936 at commit [`ef88052`](https://github.com/apache/spark/commit/ef880526e05b6143c31d98829d22d25cea659401).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #31936:
URL: https://github.com/apache/spark/pull/31936#discussion_r601501778
##########
File path: docs/running-on-yarn.md
##########
@@ -761,8 +761,27 @@ The following extra configuration options are available when the shuffle service
NodeManagers where the Spark Shuffle Service is not running.
</td>
</tr>
+<tr>
+ <td><code>spark.yarn.shuffle.service.metrics.namespace</code></td>
+ <td><code>sparkShuffleService</code></td>
+ <td>
+ The namespace to use when emitting shuffle service metrics into Hadoop metrics2 system of the
+ NodeManager.
Review comment:
great, I'm glad it works with 2.7 as well, thanks for clarifying. Yeah the concern was if it didn't work in 2.7 so I think you answered that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-806789612
overall looks good. Just to clarify did you test this on both a Hadoop < 2.9 and > 2.9 version? It sounds like you probably did but double checking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-809807994
**[Test build #136670 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136670/testReport)** for PR 31936 at commit [`ef88052`](https://github.com/apache/spark/commit/ef880526e05b6143c31d98829d22d25cea659401).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-804454370
**[Test build #136366 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136366/testReport)** for PR 31936 at commit [`37b9d14`](https://github.com/apache/spark/commit/37b9d147f28668f13e405aca952327598b52eb22).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-808354074
thanks, rekicking the build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #31936:
URL: https://github.com/apache/spark/pull/31936#discussion_r599686897
##########
File path: docs/running-on-yarn.md
##########
@@ -811,3 +830,52 @@ do the following:
to the list of filters in the <code>spark.ui.filters</code> configuration.
Be aware that the history server information may not be up-to-date with the application's state.
+
+# Running multiple versions of the Spark Shuffle Service
+
+In some cases it may be desirable to run multiple instances of the Spark Shuffle Service which are
+using different versions of Spark. This can be helpful, for example, when running a YARN cluster
+with a mixed workload of applications running multiple Spark versions, since a given version of
+the shuffle service is not always compatible with other versions of Spark. YARN versions since 2.9.0
+support the ability to run shuffle services within an isolated classloader
+(see [YARN-4577](https://issues.apache.org/jira/browse/YARN-4577)), meaning multiple Spark versions
+can coexist within a single NodeManager. The
+`yarn.nodemanager.aux-services.<service-name>.classpath` and, starting from YARN 2.10.2/3.1.1/3.2.0,
+`yarn.nodemanager.aux-services.<service-name>.remote-classpath` options can be used to configure
+this. In addition to setting up separate classpaths, it's necessary to ensure the two versions
+advertise to different ports. This can be achieved using the `spark-shuffle-site.xml` file described
+above. For example, you may have configuration like:
+
+```properties
+ yarn.nodemanager.aux-services = spark_shuffle_x,spark_shuffle_y
+ yarn.nodemanager.aux-services.spark_shuffle_x.classpath = /path/to/spark-x-yarn-shuffle.jar,/path/to/spark-x-config
+ yarn.nodemanager.aux-services.spark_shuffle_y.classpath = /path/to/spark-y-yarn-shuffle.jar,/path/to/spark-y-config
+```
+
+The two `spark-*-config` directories each contain one file, `spark-shuffle-site.xml`. These are XML
+files in the [Hadoop Configuration format](https://hadoop.apache.org/docs/r3.2.0/api/org/apache/hadoop/conf/Configuration.html)
Review comment:
Definitely!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #31936:
URL: https://github.com/apache/spark/pull/31936#discussion_r600692296
##########
File path: docs/running-on-yarn.md
##########
@@ -811,3 +831,52 @@ do the following:
to the list of filters in the <code>spark.ui.filters</code> configuration.
Be aware that the history server information may not be up-to-date with the application's state.
+
+# Running multiple versions of the Spark Shuffle Service
+
+In some cases it may be desirable to run multiple instances of the Spark Shuffle Service which are
+using different versions of Spark. This can be helpful, for example, when running a YARN cluster
+with a mixed workload of applications running multiple Spark versions, since a given version of
+the shuffle service is not always compatible with other versions of Spark. YARN versions since 2.9.0
+support the ability to run shuffle services within an isolated classloader
Review comment:
Makes sense, called this out as the first line of the section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-804621313
Thank you for pining me, @xkrogen .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-805078611
**[Test build #136417 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136417/testReport)** for PR 31936 at commit [`5e242f0`](https://github.com/apache/spark/commit/5e242f0ec236c84ea55ffd770bd418f8308e8503).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #31936:
URL: https://github.com/apache/spark/pull/31936#discussion_r600686258
##########
File path: resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnShuffleAlternateNameConfigSuite.scala
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn
+
+import java.net.URLClassLoader
+
+import org.apache.hadoop.yarn.conf.YarnConfiguration
+
+import org.apache.spark._
+import org.apache.spark.internal.config._
+import org.apache.spark.network.yarn.{YarnShuffleService, YarnTestAccessor}
+import org.apache.spark.tags.ExtendedYarnTest
+
+/**
+ * SPARK-34828: Integration test for the external shuffle service with an alternate name and
+ * configs (by using a configuration overlay)
+ */
+@ExtendedYarnTest
+class YarnShuffleAlternateNameConfigSuite extends YarnShuffleIntegrationSuite {
+
+ private[this] val shuffleServiceName = "custom_shuffle_service_name"
+
+ override def newYarnConfig(): YarnConfiguration = {
+ val yarnConfig = super.newYarnConfig()
+ yarnConfig.set(YarnConfiguration.NM_AUX_SERVICES, shuffleServiceName)
+ yarnConfig.set(YarnConfiguration.NM_AUX_SERVICE_FMT.format(shuffleServiceName),
+ classOf[YarnShuffleService].getCanonicalName)
+ val overlayConf = new YarnConfiguration()
+ // Enable authentication in the base NodeManager conf but not in the client. This would break
+ // shuffle, unless the shuffle service conf overlay overrides to turn off authentication.
+ overlayConf.setBoolean(NETWORK_AUTH_ENABLED.key, true)
+ // Add the authentication conf to a separate config object used as an overlay rather than
+ // setting it directly. This is necessary because a config overlay will override previous
+ // config overlays, but not configs which were set directly on the config object.
+ yarnConfig.addResource(overlayConf)
+ yarnConfig
+ }
+
+ override protected def extraSparkConf(): Map[String, String] =
+ super.extraSparkConf() ++ Map(SHUFFLE_SERVICE_NAME.key -> shuffleServiceName)
Review comment:
Hadoop 2.7 uses it the same way newer versions of Hadoop do, as discussed above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-809740666
**[Test build #136670 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136670/testReport)** for PR 31936 at commit [`ef88052`](https://github.com/apache/spark/commit/ef880526e05b6143c31d98829d22d25cea659401).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-809817642
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136670/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-810329701
Looks like it is clean now 🎉
@tgravescs any more concerns or are we good to go?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-810417441
Thank you, @xkrogen and @tgravescs !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-804508159
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40950/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-809705331
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136669/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-809758880
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41252/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-809817642
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136670/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-805052939
Fixed the test in `YarnShuffleServiceSuite`. I can't reproduce the failures in `YarnClusterSuite` and they seem potentially unrelated -- which specific tests failed don't seem correlated with my changes -- so I'm going to let the tests re-run and see if it was just a fluke.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #31936:
URL: https://github.com/apache/spark/pull/31936#discussion_r600691511
##########
File path: common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
##########
@@ -75,6 +76,15 @@
* is because an application running on the same Yarn cluster may choose to not use the external
* shuffle service, in which case its setting of `spark.authenticate` should be independent of
* the service's.
+ *
+ * The shuffle service will produce metrics via the YARN NodeManager's {@code metrics2} system
+ * under a namespace specified by the {@value SPARK_SHUFFLE_SERVICE_METRICS_NAMESPACE_KEY} config.
+ *
+ * By default, all configurations for the shuffle service will be taken directly from the
+ * Hadoop {@link Configuration} passed by the YARN NodeManager. It is also possible to configure
+ * the shuffle service by placing a resource named
+ * {@value SHUFFLE_SERVICE_CONF_OVERLAY_RESOURCE_NAME} into the classpath, which should be an
Review comment:
This will work on YARN 2.7 and older as well. Made this more explicit here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #31936:
URL: https://github.com/apache/spark/pull/31936#discussion_r599273586
##########
File path: docs/running-on-yarn.md
##########
@@ -761,8 +761,27 @@ The following extra configuration options are available when the shuffle service
NodeManagers where the Spark Shuffle Service is not running.
</td>
</tr>
+<tr>
+ <td><code>spark.yarn.shuffle.service.metrics.namespace</code></td>
+ <td><code>sparkShuffleService</code></td>
+ <td>
+ The namespace to use when emitting shuffle service metrics into Hadoop metrics2 system of the
+ NodeManager.
Review comment:
Could you add some description about the limitation with old Hadoop versions (like 2.7.x)? Here or at Section `Running multiple versions of the Spark Shuffle Service`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] shaneknapp commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
shaneknapp commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-809740230
test this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-809572798
**[Test build #136669 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136669/testReport)** for PR 31936 at commit [`ef88052`](https://github.com/apache/spark/commit/ef880526e05b6143c31d98829d22d25cea659401).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-808447265
Okay so the tests are still passing just fine from the module that is reporting as failed:
```
> tail -n 50 6_Build\ modules\ sql\ -\ slow\ tests\ \(JDK\ 8,\ hadoop3.2,\ hive2.3\).txt | head
2021-03-26T17:28:34.3588275Z [info] ScalaTest
2021-03-26T17:28:34.3593568Z [info] Run completed in 38 minutes, 4 seconds.
2021-03-26T17:28:34.3596067Z [info] Total number of tests run: 1062
2021-03-26T17:28:34.3598550Z [info] Suites: completed 423, aborted 0
2021-03-26T17:28:34.3601246Z [info] Tests: succeeded 1062, failed 0, canceled 0, ignored 1, pending 0
2021-03-26T17:28:34.3603891Z [info] All tests passed.
2021-03-26T17:28:34.3660549Z [info] Passed: Total 1062, Failed 0, Errors 0, Passed 1062, Ignored 1
2021-03-26T17:28:45.5978270Z [warn] In the last 2325 seconds, 9.286 (0.4%) were spent in GC. [Heap: 2.41GB free of 3.00GB, max 3.56GB] Consider increasing the JVM heap using `-Xmx` or try a different collector, e.g. `-XX:+UseG1GC`, for better performance.
2021-03-26T17:28:45.6688003Z [success] Total time: 2326 s (38:46), completed Mar 26, 2021 5:28:45 PM
2021-03-26T17:28:46.6311617Z
```
But I did notice that there are compilation errors:
```
[error] /home/runner/work/spark/spark/mllib/target/java/org/apache/spark/mllib/util/MLlibTestSparkContext.java:10:1: error: illegal combination of modifiers: public and protected
[error] protected class testImplicits {
[error] ^
[error] /home/runner/work/spark/spark/sql/core/target/java/org/apache/spark/sql/UDFSuite.java:4:1: error: modifier static not allowed here
[error] static public class MalformedNonPrimitiveFunction implements scala.Function1<java.lang.String, java.lang.Object>, scala.Serializable {
[error] ^
[error] /home/runner/work/spark/spark/sql/core/target/java/org/apache/spark/sql/test/SQLTestUtilsBase.java:19:1: error: illegal combination of modifiers: public and protected
[error] protected class testImplicits {
[error] ^
```
(these are about 100 of these)
But none of them are related to my changes. I'm pretty confused about why they're showing up here, do you have any insight?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-804509954
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136366/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-806087013
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136477/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-804454745
cc @mridulm @HeartSaVioR @tgravescs @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-804496094
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40950/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-804902211
this was tried once before under https://github.com/apache/spark/pull/26000 so just putting it here for reference. I think the main thing is if we are going to support that we make sure its truly configurable and not just half done. But it sounds like you are trying to address the concerns brought up there but I need to do a more detailed review.
One thing doing a quick skim I don't see if in YarnShuffleService.java its still registering the AuxiliaryService with the name spark_shuffle, Is that not really used by the yarn auxiliary service and it uses what you configure in the node manager? In the very least that needs to be documented in the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #31936:
URL: https://github.com/apache/spark/pull/31936#discussion_r600685042
##########
File path: docs/running-on-yarn.md
##########
@@ -761,8 +761,27 @@ The following extra configuration options are available when the shuffle service
NodeManagers where the Spark Shuffle Service is not running.
</td>
</tr>
+<tr>
+ <td><code>spark.yarn.shuffle.service.metrics.namespace</code></td>
+ <td><code>sparkShuffleService</code></td>
+ <td>
+ The namespace to use when emitting shuffle service metrics into Hadoop metrics2 system of the
+ NodeManager.
Review comment:
> it looks like the name referenced by the node manager works with the Hadoop 2.9+ custom class loader, but I assume with Hadoop 2.7 it requires the spark_shuffle name ? hence the spark.shuffle.service.name won't work unless you have recompiled the code and manually changed it.
No, this is not correct. YARN ignores the hard-coded name on _all_ versions of YARN. Take a look at `AuxServices` on the 2.7.0 branch:
https://github.com/apache/hadoop/blob/f95b390df2ca7d599f0ad82cf6e8d980469e7abb/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/AuxServices.java#L129-L136
`spark.shuffle.service.name` works fine on Hadoop 2.7, it is only the isolated classloader that won't work on older versions.
> I'm worrying about the situation some users try to use `Apache Spark distribution (with Hadoop 2.7)` at YARN 2.9+ cluster. Does it work?
I don't quite understand the concern here. Does my explanation above address your question? We haven't changed any of the interfaces used to interact with YARN, there should be no binary compatibility issues or anything of that sort. I can test whichever combination of `Spark Version + Hadoop Version Distribution` running on top of `Hadoop Version YARN` you like, but I am failing to see where the concern is / what you'd like me to look for.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-806265869
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41061/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-810346449
Fantastic, many thanks @tgravescs !
Also thanks to @dongjoon-hyun for some early feedback!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-809722198
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41251/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-809572798
**[Test build #136669 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136669/testReport)** for PR 31936 at commit [`ef88052`](https://github.com/apache/spark/commit/ef880526e05b6143c31d98829d22d25cea659401).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-809704199
**[Test build #136669 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136669/testReport)** for PR 31936 at commit [`ef88052`](https://github.com/apache/spark/commit/ef880526e05b6143c31d98829d22d25cea659401).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-806236664
Only test failure from the last run was `org.apache.spark.scheduler.DAGSchedulerSuite`, I don't think it looks related to my changes. This test also succeeded on a previous run and the only change since then was documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-806007849
Updated documentation to make it a bit more clear which features need YARN >= 2.9.0 (only the isolated classloader / multiple shuffle service support, none of the other new additions).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-804509954
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136366/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-806255211
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41061/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-809730491
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41251/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #31936:
URL: https://github.com/apache/spark/pull/31936#discussion_r600183889
##########
File path: docs/running-on-yarn.md
##########
@@ -761,8 +761,27 @@ The following extra configuration options are available when the shuffle service
NodeManagers where the Spark Shuffle Service is not running.
</td>
</tr>
+<tr>
+ <td><code>spark.yarn.shuffle.service.metrics.namespace</code></td>
+ <td><code>sparkShuffleService</code></td>
+ <td>
+ The namespace to use when emitting shuffle service metrics into Hadoop metrics2 system of the
+ NodeManager.
Review comment:
I'm worrying about the situation some users try to use `Apache Spark distribution (with Hadoop 2.7)` at YARN 2.9+ cluster. Does it work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-806237871
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41061/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-804491661
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40950/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-804509371
**[Test build #136366 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136366/testReport)** for PR 31936 at commit [`37b9d14`](https://github.com/apache/spark/commit/37b9d147f28668f13e405aca952327598b52eb22).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #31936:
URL: https://github.com/apache/spark/pull/31936#discussion_r599271678
##########
File path: docs/running-on-yarn.md
##########
@@ -811,3 +830,52 @@ do the following:
to the list of filters in the <code>spark.ui.filters</code> configuration.
Be aware that the history server information may not be up-to-date with the application's state.
+
+# Running multiple versions of the Spark Shuffle Service
+
+In some cases it may be desirable to run multiple instances of the Spark Shuffle Service which are
+using different versions of Spark. This can be helpful, for example, when running a YARN cluster
+with a mixed workload of applications running multiple Spark versions, since a given version of
+the shuffle service is not always compatible with other versions of Spark. YARN versions since 2.9.0
+support the ability to run shuffle services within an isolated classloader
+(see [YARN-4577](https://issues.apache.org/jira/browse/YARN-4577)), meaning multiple Spark versions
+can coexist within a single NodeManager. The
+`yarn.nodemanager.aux-services.<service-name>.classpath` and, starting from YARN 2.10.2/3.1.1/3.2.0,
+`yarn.nodemanager.aux-services.<service-name>.remote-classpath` options can be used to configure
+this. In addition to setting up separate classpaths, it's necessary to ensure the two versions
+advertise to different ports. This can be achieved using the `spark-shuffle-site.xml` file described
+above. For example, you may have configuration like:
+
+```properties
+ yarn.nodemanager.aux-services = spark_shuffle_x,spark_shuffle_y
+ yarn.nodemanager.aux-services.spark_shuffle_x.classpath = /path/to/spark-x-yarn-shuffle.jar,/path/to/spark-x-config
+ yarn.nodemanager.aux-services.spark_shuffle_y.classpath = /path/to/spark-y-yarn-shuffle.jar,/path/to/spark-y-config
+```
+
+The two `spark-*-config` directories each contain one file, `spark-shuffle-site.xml`. These are XML
+files in the [Hadoop Configuration format](https://hadoop.apache.org/docs/r3.2.0/api/org/apache/hadoop/conf/Configuration.html)
Review comment:
Shall we reference Apache Hadoop 3.2.2 doc instead of 3.2.0 because we are using Apache Spark 3.2.2?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-806087013
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136477/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-807168081
I went through the logs for the [most recent test failure](https://github.com/apache/spark/pull/31936/checks?check_run_id=2193694302) and couldn't find any test failures:
```
> grep -E -e "completed.+aborted" -e "Passed.+Failed" 6_Build\ modules\ sql\ -\ slow\ tests\ \(JDK\ 8,\ hadoop3.2,\ hive2.3\).txt
2021-03-25T15:45:53.5726513Z [info] Suites: completed 423, aborted 0
2021-03-25T15:45:53.5834071Z [info] Passed: Total 1062, Failed 0, Errors 0, Passed 1062, Ignored 1
```
(I had to download the logs since they were too big for the GitHub web viewer).
Not sure what's wrong but this same suite passed on my previous build with only documentation changes since then (the previous failure was in a different module).
> Just to clarify did you test this on both a Hadoop < 2.9 and > 2.9 version? It sounds like you probably did but double checking.
@tgravescs -- I think I did test on a Hadoop 2.7 cluster previously, but just to be sure, I tried just now. Works great using a custom shuffle service name and conf overlay via `spark-shuffle-site.xml`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-809556421
Thanks for the pointer @tgravescs ! Looks like a new build was triggered (but not yet run) and I can't find a way to get the old logs so I can't check in more detail why those ones failed.
I tried to reproduce those failures locally after a rebase and haven't been able to. I'm going to push again with the new rebase and see if picking up recent changes from `master` resolves things. If not I'll check the logs and dig in more detail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on server side
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31936:
URL: https://github.com/apache/spark/pull/31936#issuecomment-809705331
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136669/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org