[jira] [Commented] (SPARK-1267) Add a pip installer for PySpark

["Alex Gaudio (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-1267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14083355#comment-14083355 ] 

Alex Gaudio commented on SPARK-1267:
------------------------------------

I'm all for pip installable pyspark, but I'm confused about the ideal way to install the pyspark code.  I'd also prefer to avoid introducing an extra variable, SPARK_VERSION.  It seems to me that if we had the typical setup.py file that downloaded code from PyPi, then users would have to deal with differences in dependencies between the python version in PyPi and in their code pointed to by SPARK_HOME.   Additionally, users would still need to download the spark jars or set SPARK_HOME, which means two (possibly different) versions of the python code are flying around. The fact that users have to manage the version, download spark into SPARK_HOME, and pip install pyspark doesn't seem quite right.

What do you think about this:  We create a setup.py file that requires SPARK_HOME be set in the environment (requiring that the user have downloaded Spark) BEFORE the pyspark code gets installed.  

An additional idea we could consider:  Then, when pip or a user calls pyspark, we have "python setup.py install" redirect to "python setup.py develop."  This installs pyspark in "development mode" and means that the pyspark code pointed to by $SPARK_HOME/python is the source of truth.  (more about development mode here: https://pythonhosted.org/setuptools/setuptools.html#development-mode).  My thinking for this is that since users need to specify SPARK_HOME, we might as well keep the python library with the spark code (as it currently is) to avoid potential compatibility conflicts.  As a maintainer, we also don't need to update PyPi with the latest version of pyspark.  Using "develop mode" as default may be a bad idea.  I also don't know how to automatically prefer "setup.py develop" over "setup.py install".

Last, and perhaps most obvious, if we create a setup.py file, we could also probably no longer include the py4j egg in the spark downloads as we'd rely on setuptools to provide the external libraries.



> Add a pip installer for PySpark
> -------------------------------
>
>                 Key: SPARK-1267
>                 URL: https://issues.apache.org/jira/browse/SPARK-1267
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark
>            Reporter: Prabin Banka
>            Priority: Minor
>              Labels: pyspark
>
> Please refer to this mail archive,
> http://mail-archives.apache.org/mod_mbox/spark-user/201311.mbox/%3CCAOEPXP7jKiw-3M8eh2giBcs8gEkZ1upHpGb=FqOUcVSCYwjhNg@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.2#6252)