git commit: SPARK-1404: Always upgrade spark-env.sh vars to environment vars

[pw...@apache.org]

Repository: spark
Updated Branches:
  refs/heads/master 7f32fd42a -> 01cf4c402


SPARK-1404: Always upgrade spark-env.sh vars to environment vars

This was broken when spark-env.sh was made idempotent, as the idempotence check is an environment variable, but the spark-env.sh variables may not have been.

Tested in zsh, bash, and sh.

Author: Aaron Davidson <aa...@databricks.com>

Closes #310 from aarondav/SPARK-1404 and squashes the following commits:

c3406a5 [Aaron Davidson] Add extra export in spark-shell
6a0e340 [Aaron Davidson] SPARK-1404: Always upgrade spark-env.sh vars to environment vars


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/01cf4c40
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/01cf4c40
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/01cf4c40

Branch: refs/heads/master
Commit: 01cf4c402b9fda59680e56112bfaa2b748416d0e
Parents: 7f32fd4
Author: Aaron Davidson <aa...@databricks.com>
Authored: Fri Apr 4 09:50:24 2014 -0700
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Fri Apr 4 09:50:24 2014 -0700

----------------------------------------------------------------------
 bin/load-spark-env.sh | 3 +++
 bin/spark-shell       | 4 ++--
 2 files changed, 5 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/01cf4c40/bin/load-spark-env.sh
----------------------------------------------------------------------
diff --git a/bin/load-spark-env.sh b/bin/load-spark-env.sh
index 476dd82..d425f9f 100644
--- a/bin/load-spark-env.sh
+++ b/bin/load-spark-env.sh
@@ -30,6 +30,9 @@ if [ -z "$SPARK_ENV_LOADED" ]; then
   use_conf_dir=${SPARK_CONF_DIR:-"$parent_dir/conf"}
 
   if [ -f "${use_conf_dir}/spark-env.sh" ]; then
+    # Promote all variable declarations to environment (exported) variables
+    set -a
     . "${use_conf_dir}/spark-env.sh"
+    set +a
   fi
 fi

http://git-wip-us.apache.org/repos/asf/spark/blob/01cf4c40/bin/spark-shell
----------------------------------------------------------------------
diff --git a/bin/spark-shell b/bin/spark-shell
index fac006c..535ee3c 100755
--- a/bin/spark-shell
+++ b/bin/spark-shell
@@ -127,7 +127,7 @@ function set_spark_log_conf(){
 
 function set_spark_master(){
     if ! [[ "$1" =~ $ARG_FLAG_PATTERN ]]; then
-        MASTER="$1"
+        export MASTER="$1"
     else
         out_error "wrong format for $2"
     fi
@@ -145,7 +145,7 @@ function resolve_spark_master(){
     fi
 
     if [ -z "$MASTER" ]; then
-        MASTER="$DEFAULT_MASTER"
+        export MASTER="$DEFAULT_MASTER"
     fi
 
 }