You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by td...@apache.org on 2014/05/23 05:32:35 UTC
git commit: [SPARK-1896] Respect spark.master (and --master) before
MASTER in spark-shell
Repository: spark
Updated Branches:
refs/heads/master 8edbee7d1 -> cce77457e
[SPARK-1896] Respect spark.master (and --master) before MASTER in spark-shell
The hierarchy for configuring the Spark master in the shell is as follows:
```
MASTER > --master > spark.master (spark-defaults.conf)
```
This is inconsistent with the way we run normal applications, which is:
```
--master > spark.master (spark-defaults.conf) > MASTER
```
I was trying to run a shell locally on a standalone cluster launched through the ec2 scripts, which automatically set `MASTER` in spark-env.sh. It was surprising to me that `--master` didn't take effect, considering that this is the way we tell users to set their masters [here](http://people.apache.org/~pwendell/spark-1.0.0-rc7-docs/scala-programming-guide.html#initializing-spark).
Author: Andrew Or <an...@gmail.com>
Closes #846 from andrewor14/shell-master and squashes the following commits:
2cb81c9 [Andrew Or] Respect spark.master before MASTER in REPL
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/cce77457
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/cce77457
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/cce77457
Branch: refs/heads/master
Commit: cce77457e00aa5f1f4db3d50454cf257efb156ed
Parents: 8edbee7
Author: Andrew Or <an...@gmail.com>
Authored: Thu May 22 20:32:27 2014 -0700
Committer: Tathagata Das <ta...@gmail.com>
Committed: Thu May 22 20:32:27 2014 -0700
----------------------------------------------------------------------
repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/cce77457/repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
----------------------------------------------------------------------
diff --git a/repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala b/repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
index 55684e9..5f34362 100644
--- a/repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
+++ b/repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
@@ -962,11 +962,10 @@ class SparkILoop(in0: Option[BufferedReader], protected val out: JPrintWriter,
private def getMaster(): String = {
val master = this.master match {
case Some(m) => m
- case None => {
+ case None =>
val envMaster = sys.env.get("MASTER")
val propMaster = sys.props.get("spark.master")
- envMaster.orElse(propMaster).getOrElse("local[*]")
- }
+ propMaster.orElse(envMaster).getOrElse("local[*]")
}
master
}