You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2017/11/09 05:33:12 UTC
spark git commit: [SPARK-22466][SPARK SUBMIT] export SPARK_CONF_DIR
while conf is default
Repository: spark
Updated Branches:
refs/heads/master 6447d7bc1 -> ee571d79e
[SPARK-22466][SPARK SUBMIT] export SPARK_CONF_DIR while conf is default
## What changes were proposed in this pull request?
We use SPARK_CONF_DIR to switch spark conf directory and can be visited if we explicitly export it in spark-env.sh, but with default settings, it can't be done. This PR export SPARK_CONF_DIR while it is default.
### Before
```
KentKentsMacBookPro ~/Documents/spark-packages/spark-2.3.0-SNAPSHOT-bin-master bin/spark-shell --master local
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
17/11/08 10:28:44 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/11/08 10:28:45 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
Spark context Web UI available at http://169.254.168.63:4041
Spark context available as 'sc' (master = local, app id = local-1510108125770).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.3.0-SNAPSHOT
/_/
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_65)
Type in expressions to have them evaluated.
Type :help for more information.
scala> sys.env.get("SPARK_CONF_DIR")
res0: Option[String] = None
```
### After
```
scala> sys.env.get("SPARK_CONF_DIR")
res0: Option[String] = Some(/Users/Kent/Documents/spark/conf)
```
## How was this patch tested?
vanzin
Author: Kent Yao <ya...@hotmail.com>
Closes #19688 from yaooqinn/SPARK-22466.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/ee571d79
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/ee571d79
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/ee571d79
Branch: refs/heads/master
Commit: ee571d79e52dc7e0ad7ae80619c12a5c0b90b5a5
Parents: 6447d7b
Author: Kent Yao <ya...@hotmail.com>
Authored: Thu Nov 9 14:33:08 2017 +0900
Committer: hyukjinkwon <gu...@gmail.com>
Committed: Thu Nov 9 14:33:08 2017 +0900
----------------------------------------------------------------------
bin/load-spark-env.cmd | 12 +++++-------
bin/load-spark-env.sh | 9 +++------
conf/spark-env.sh.template | 3 ++-
3 files changed, 10 insertions(+), 14 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/ee571d79/bin/load-spark-env.cmd
----------------------------------------------------------------------
diff --git a/bin/load-spark-env.cmd b/bin/load-spark-env.cmd
index f946197..cefa513 100644
--- a/bin/load-spark-env.cmd
+++ b/bin/load-spark-env.cmd
@@ -19,15 +19,13 @@ rem
rem This script loads spark-env.cmd if it exists, and ensures it is only loaded once.
rem spark-env.cmd is loaded from SPARK_CONF_DIR if set, or within the current directory's
-rem conf/ subdirectory.
+rem conf\ subdirectory.
if [%SPARK_ENV_LOADED%] == [] (
set SPARK_ENV_LOADED=1
- if not [%SPARK_CONF_DIR%] == [] (
- set user_conf_dir=%SPARK_CONF_DIR%
- ) else (
- set user_conf_dir=..\conf
+ if [%SPARK_CONF_DIR%] == [] (
+ set SPARK_CONF_DIR=%~dp0..\conf
)
call :LoadSparkEnv
@@ -54,6 +52,6 @@ if [%SPARK_SCALA_VERSION%] == [] (
exit /b 0
:LoadSparkEnv
-if exist "%user_conf_dir%\spark-env.cmd" (
- call "%user_conf_dir%\spark-env.cmd"
+if exist "%SPARK_CONF_DIR%\spark-env.cmd" (
+ call "%SPARK_CONF_DIR%\spark-env.cmd"
)
http://git-wip-us.apache.org/repos/asf/spark/blob/ee571d79/bin/load-spark-env.sh
----------------------------------------------------------------------
diff --git a/bin/load-spark-env.sh b/bin/load-spark-env.sh
index d05d94e..0b5006d 100644
--- a/bin/load-spark-env.sh
+++ b/bin/load-spark-env.sh
@@ -29,15 +29,12 @@ fi
if [ -z "$SPARK_ENV_LOADED" ]; then
export SPARK_ENV_LOADED=1
- # Returns the parent of the directory this script lives in.
- parent_dir="${SPARK_HOME}"
+ export SPARK_CONF_DIR="${SPARK_CONF_DIR:-"${SPARK_HOME}"/conf}"
- user_conf_dir="${SPARK_CONF_DIR:-"$parent_dir"/conf}"
-
- if [ -f "${user_conf_dir}/spark-env.sh" ]; then
+ if [ -f "${SPARK_CONF_DIR}/spark-env.sh" ]; then
# Promote all variable declarations to environment (exported) variables
set -a
- . "${user_conf_dir}/spark-env.sh"
+ . "${SPARK_CONF_DIR}/spark-env.sh"
set +a
fi
fi
http://git-wip-us.apache.org/repos/asf/spark/blob/ee571d79/conf/spark-env.sh.template
----------------------------------------------------------------------
diff --git a/conf/spark-env.sh.template b/conf/spark-env.sh.template
index f8c895f..bc92c78 100755
--- a/conf/spark-env.sh.template
+++ b/conf/spark-env.sh.template
@@ -32,7 +32,8 @@
# - SPARK_LOCAL_DIRS, storage directories to use on this node for shuffle and RDD data
# - MESOS_NATIVE_JAVA_LIBRARY, to point to your libmesos.so if you use Mesos
-# Options read in YARN client mode
+# Options read in YARN client/cluster mode
+# - SPARK_CONF_DIR, Alternate conf dir. (Default: ${SPARK_HOME}/conf)
# - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files
# - YARN_CONF_DIR, to point Spark towards YARN configuration files when you use YARN
# - SPARK_EXECUTOR_CORES, Number of cores for the executors (Default: 1).
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org