You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2017/06/09 08:26:34 UTC
spark git commit: [SPARK-20995][CORE] Spark-env.sh.template' should
add 'YARN_CONF_DIR' configuration instructions.
Repository: spark
Updated Branches:
refs/heads/master 5a3371883 -> bdcd6e4c6
[SPARK-20995][CORE] Spark-env.sh.template' should add 'YARN_CONF_DIR' configuration instructions.
## What changes were proposed in this pull request?
Ensure that `HADOOP_CONF_DIR` or `YARN_CONF_DIR` points to the directory which contains the (client side) configuration files for the Hadoop cluster.
These configs are used to write to HDFS and connect to the YARN ResourceManager. The
configuration contained in this directory will be distributed to the YARN cluster so that all
containers used by the application use the same configuration.
Sometimes, `HADOOP_CONF_DIR` is set to the hdfs configuration file path. So, YARN_CONF_DIR should be set to the yarn configuration file path.
My project configuration item of 'spark-env.sh ' is as follows:
'HADOOP_CONF_DIR' configuration file path. List the relevant documents below:
'YARN_CONF_DIR' configuration file path. List the relevant documents below:
So, 'Spark-env.sh.template' should add 'YARN_CONF_DIR' configuration instructions.
## How was this patch tested?
manual tests
Please review http://spark.apache.org/contributing.html before opening a pull request.
Author: guoxiaolong <gu...@zte.com.cn>
Author: 郭小龙 10207633 <gu...@zte.com.cn>
Author: guoxiaolongzte <gu...@zte.com.cn>
Closes #18212 from guoxiaolongzte/SPARK-20995.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/bdcd6e4c
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/bdcd6e4c
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/bdcd6e4c
Branch: refs/heads/master
Commit: bdcd6e4c680ebd3ddf5c1baaeba31134b143dfb4
Parents: 5a33718
Author: guoxiaolong <gu...@zte.com.cn>
Authored: Fri Jun 9 09:26:30 2017 +0100
Committer: Sean Owen <so...@cloudera.com>
Committed: Fri Jun 9 09:26:30 2017 +0100
----------------------------------------------------------------------
conf/spark-env.sh.template | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/bdcd6e4c/conf/spark-env.sh.template
----------------------------------------------------------------------
diff --git a/conf/spark-env.sh.template b/conf/spark-env.sh.template
index b7c985a..b9aab5a 100755
--- a/conf/spark-env.sh.template
+++ b/conf/spark-env.sh.template
@@ -34,6 +34,7 @@
# Options read in YARN client mode
# - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files
+# - YARN_CONF_DIR, to point Spark towards YARN configuration files when you use YARN
# - SPARK_EXECUTOR_CORES, Number of cores for the executors (Default: 1).
# - SPARK_EXECUTOR_MEMORY, Memory per Executor (e.g. 1000M, 2G) (Default: 1G)
# - SPARK_DRIVER_MEMORY, Memory for Driver (e.g. 1000M, 2G) (Default: 1G)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org