You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by we...@apache.org on 2017/06/24 05:23:48 UTC
spark git commit: [SPARK-21159][CORE] Don't try to connect to
launcher in standalone cluster mode.
Repository: spark
Updated Branches:
refs/heads/master b837bf9ae -> bfd73a7c4
[SPARK-21159][CORE] Don't try to connect to launcher in standalone cluster mode.
Monitoring for standalone cluster mode is not implemented (see SPARK-11033), but
the same scheduler implementation is used, and if it tries to connect to the
launcher it will fail. So fix the scheduler so it only tries that in client mode;
cluster mode applications will be correctly launched and will work, but monitoring
through the launcher handle will not be available.
Tested by running a cluster mode app with "SparkLauncher.startApplication".
Author: Marcelo Vanzin <va...@cloudera.com>
Closes #18397 from vanzin/SPARK-21159.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/bfd73a7c
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/bfd73a7c
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/bfd73a7c
Branch: refs/heads/master
Commit: bfd73a7c48b87456d1b84d826e04eca938a1be64
Parents: b837bf9
Author: Marcelo Vanzin <va...@cloudera.com>
Authored: Sat Jun 24 13:23:43 2017 +0800
Committer: Wenchen Fan <we...@databricks.com>
Committed: Sat Jun 24 13:23:43 2017 +0800
----------------------------------------------------------------------
.../spark/scheduler/cluster/StandaloneSchedulerBackend.scala | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/bfd73a7c/core/src/main/scala/org/apache/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/scheduler/cluster/StandaloneSchedulerBackend.scala b/core/src/main/scala/org/apache/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
index fd8e644..a4e2a74 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
@@ -58,7 +58,13 @@ private[spark] class StandaloneSchedulerBackend(
override def start() {
super.start()
- launcherBackend.connect()
+
+ // SPARK-21159. The scheduler backend should only try to connect to the launcher when in client
+ // mode. In cluster mode, the code that submits the application to the Master needs to connect
+ // to the launcher instead.
+ if (sc.deployMode == "client") {
+ launcherBackend.connect()
+ }
// The endpoint for executors to talk to us
val driverUrl = RpcEndpointAddress(
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org