You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by va...@apache.org on 2018/05/01 15:28:29 UTC
spark git commit: [SPARK-23941][MESOS] Mesos task failed on specific
spark app name
Repository: spark
Updated Branches:
refs/heads/master 7bbec0dce -> 6782359a0
[SPARK-23941][MESOS] Mesos task failed on specific spark app name
## What changes were proposed in this pull request?
Shell escaped the name passed to spark-submit and change how conf attributes are shell escaped.
## How was this patch tested?
This test has been tested manually with Hive-on-spark with mesos or with the use case described in the issue with the sparkPi application with a custom name which contains illegal shell characters.
With this PR, hive-on-spark on mesos works like a charm with hive 3.0.0-SNAPSHOT.
I state that this contribution is my original work and that I license the work to the project under the project’s open source license
Author: Bounkong Khamphousone <bo...@ebiznext.com>
Closes #21014 from tiboun/fix/SPARK-23941.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6782359a
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/6782359a
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/6782359a
Branch: refs/heads/master
Commit: 6782359a04356e4cde32940861bf2410ef37f445
Parents: 7bbec0d
Author: Bounkong Khamphousone <bo...@ebiznext.com>
Authored: Tue May 1 08:28:21 2018 -0700
Committer: Marcelo Vanzin <va...@cloudera.com>
Committed: Tue May 1 08:28:21 2018 -0700
----------------------------------------------------------------------
.../spark/scheduler/cluster/mesos/MesosClusterScheduler.scala | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/6782359a/resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
----------------------------------------------------------------------
diff --git a/resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala b/resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
index d224a73..b36f464 100644
--- a/resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
+++ b/resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
@@ -530,9 +530,9 @@ private[spark] class MesosClusterScheduler(
.filter { case (key, _) => !replicatedOptionsBlacklist.contains(key) }
.toMap
(defaultConf ++ driverConf).foreach { case (key, value) =>
- options ++= Seq("--conf", s""""$key=${shellEscape(value)}"""".stripMargin) }
+ options ++= Seq("--conf", s"${key}=${value}") }
- options
+ options.map(shellEscape)
}
/**
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org