You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2016/09/01 08:22:22 UTC
[jira] [Commented] (SPARK-17336) Repeated calls
sbin/spark-config.sh file Causes ${PYTHONPATH} Value duplicate
[ https://issues.apache.org/jira/browse/SPARK-17336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15454745#comment-15454745 ]
Sean Owen commented on SPARK-17336:
-----------------------------------
[~axu4apache] I assume you modified the scripts to print PYTHONPATH in order to show this behavior.
Yeah it looks like the file that sets it is repeatedly sourced so this path is appended many times.
I don't know of a cleaner way to do this other than to see if the variable already contains the path and only append if it doesn't. Do you want to try that in a PR?
> Repeated calls sbin/spark-config.sh file Causes ${PYTHONPATH} Value duplicate
> -----------------------------------------------------------------------------
>
> Key: SPARK-17336
> URL: https://issues.apache.org/jira/browse/SPARK-17336
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 2.0.0
> Reporter: anxu
>
> On Spark start up by command: sbin/start-all.sh, the sbin/spark-config.sh Repeated calls. In sbin/spark-config.sh code.
> {code:title=sbin/spark-config.sh|borderStyle=solid}
> # Add the PySpark classes to the PYTHONPATH:
> export PYTHONPATH="${SPARK_HOME}/python:${PYTHONPATH}"
> export PYTHONPATH="${SPARK_HOME}/python/lib/py4j-0.10.3-src.zip:${PYTHONPATH}"
> {code}
> {color:red}PYTHONPATH{color} has duplicate Value.
> example:
> {code:borderStyle=solid}
> axu4iMac:spark-2.0.0-hadoop2.4 axu$ sbin/start-all.sh | grep PYTHONPATH
> axu.print [Log] [6,16,31] [sbin/spark-config.sh] 定义PYTHONPATH
> axu.print [sbin/spark-config.sh] [Define Global] PYTHONPATH(1): [/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:]
> axu.print [Log] [7,17,32] [sbin/spark-config.sh] 再次定义PYTHONPATH
> axu.print [sbin/spark-config.sh] [Define Global] PYTHONPATH(2): [/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:]
> axu.print [Log] [6,16,31] [sbin/spark-config.sh] 定义PYTHONPATH
> axu.print [sbin/spark-config.sh] [Define Global] PYTHONPATH(1): [/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:]
> axu.print [Log] [7,17,32] [sbin/spark-config.sh] 再次定义PYTHONPATH
> axu.print [sbin/spark-config.sh] [Define Global] PYTHONPATH(2): [/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:]
> axu.print [Log] [6,16,31] [sbin/spark-config.sh] 定义PYTHONPATH
> axu.print [sbin/spark-config.sh] [Define Global] PYTHONPATH(1): [/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:]
> axu.print [Log] [7,17,32] [sbin/spark-config.sh] 再次定义PYTHONPATH
> axu.print [sbin/spark-config.sh] [Define Global] PYTHONPATH(2): [/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python/lib/py4j-0.10.1-src.zip:/Users/axu/code/axuProject/spark-2.0.0-hadoop2.4/python:]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org