You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2016/09/11 09:19:44 UTC
spark git commit: [SPARK-17336][PYSPARK] Fix appending multiple times
to PYTHONPATH from spark-config.sh
Repository: spark
Updated Branches:
refs/heads/master bf2221737 -> c76baff0c
[SPARK-17336][PYSPARK] Fix appending multiple times to PYTHONPATH from spark-config.sh
## What changes were proposed in this pull request?
During startup of Spark standalone, the script file spark-config.sh appends to the PYTHONPATH and can be sourced many times, causing duplicates in the path. This change adds a env flag that is set when the PYTHONPATH is appended so it will happen only one time.
## How was this patch tested?
Manually started standalone master/worker and verified PYTHONPATH has no duplicate entries.
Author: Bryan Cutler <cu...@gmail.com>
Closes #15028 from BryanCutler/fix-duplicate-pythonpath-SPARK-17336.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c76baff0
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c76baff0
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c76baff0
Branch: refs/heads/master
Commit: c76baff0cc4775c2191d075cc9a8176e4915fec8
Parents: bf22217
Author: Bryan Cutler <cu...@gmail.com>
Authored: Sun Sep 11 10:19:39 2016 +0100
Committer: Sean Owen <so...@cloudera.com>
Committed: Sun Sep 11 10:19:39 2016 +0100
----------------------------------------------------------------------
sbin/spark-config.sh | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/c76baff0/sbin/spark-config.sh
----------------------------------------------------------------------
diff --git a/sbin/spark-config.sh b/sbin/spark-config.sh
index a7a44cd..b728448 100755
--- a/sbin/spark-config.sh
+++ b/sbin/spark-config.sh
@@ -26,5 +26,8 @@ fi
export SPARK_CONF_DIR="${SPARK_CONF_DIR:-"${SPARK_HOME}/conf"}"
# Add the PySpark classes to the PYTHONPATH:
-export PYTHONPATH="${SPARK_HOME}/python:${PYTHONPATH}"
-export PYTHONPATH="${SPARK_HOME}/python/lib/py4j-0.10.3-src.zip:${PYTHONPATH}"
+if [ -z "${PYSPARK_PYTHONPATH_SET}" ]; then
+ export PYTHONPATH="${SPARK_HOME}/python:${PYTHONPATH}"
+ export PYTHONPATH="${SPARK_HOME}/python/lib/py4j-0.10.3-src.zip:${PYTHONPATH}"
+ export PYSPARK_PYTHONPATH_SET=1
+fi
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org