You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by jo...@apache.org on 2016/11/16 22:22:19 UTC
spark git commit: [SPARK-1267][SPARK-18129] Allow PySpark to be pip
installed
Repository: spark
Updated Branches:
refs/heads/master bb6cdfd9a -> a36a76ac4
[SPARK-1267][SPARK-18129] Allow PySpark to be pip installed
## What changes were proposed in this pull request?
This PR aims to provide a pip installable PySpark package. This does a bunch of work to copy the jars over and package them with the Python code (to prevent challenges from trying to use different versions of the Python code with different versions of the JAR). It does not currently publish to PyPI but that is the natural follow up (SPARK-18129).
Done:
- pip installable on conda [manual tested]
- setup.py installed on a non-pip managed system (RHEL) with YARN [manual tested]
- Automated testing of this (virtualenv)
- packaging and signing with release-build*
Possible follow up work:
- release-build update to publish to PyPI (SPARK-18128)
- figure out who owns the pyspark package name on prod PyPI (is it someone with in the project or should we ask PyPI or should we choose a different name to publish with like ApachePySpark?)
- Windows support and or testing ( SPARK-18136 )
- investigate details of wheel caching and see if we can avoid cleaning the wheel cache during our test
- consider how we want to number our dev/snapshot versions
Explicitly out of scope:
- Using pip installed PySpark to start a standalone cluster
- Using pip installed PySpark for non-Python Spark programs
*I've done some work to test release-build locally but as a non-committer I've just done local testing.
## How was this patch tested?
Automated testing with virtualenv, manual testing with conda, a system wide install, and YARN integration.
release-build changes tested locally as a non-committer (no testing of upload artifacts to Apache staging websites)
Author: Holden Karau <ho...@us.ibm.com>
Author: Juliet Hougland <ju...@cloudera.com>
Author: Juliet Hougland <no...@myemail.com>
Closes #15659 from holdenk/SPARK-1267-pip-install-pyspark.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a36a76ac
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/a36a76ac
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/a36a76ac
Branch: refs/heads/master
Commit: a36a76ac43c36a3b897a748bd9f138b629dbc684
Parents: bb6cdfd
Author: Holden Karau <ho...@us.ibm.com>
Authored: Wed Nov 16 14:22:15 2016 -0800
Committer: Josh Rosen <jo...@databricks.com>
Committed: Wed Nov 16 14:22:15 2016 -0800
----------------------------------------------------------------------
.gitignore | 2 +
bin/beeline | 2 +-
bin/find-spark-home | 41 ++++
bin/load-spark-env.sh | 2 +-
bin/pyspark | 6 +-
bin/run-example | 2 +-
bin/spark-class | 6 +-
bin/spark-shell | 4 +-
bin/spark-sql | 2 +-
bin/spark-submit | 2 +-
bin/sparkR | 2 +-
dev/create-release/release-build.sh | 26 ++-
dev/create-release/release-tag.sh | 11 +-
dev/lint-python | 4 +-
dev/make-distribution.sh | 16 +-
dev/pip-sanity-check.py | 36 ++++
dev/run-pip-tests | 115 ++++++++++
dev/run-tests-jenkins.py | 1 +
dev/run-tests.py | 7 +
dev/sparktestsupport/__init__.py | 1 +
docs/building-spark.md | 8 +
docs/index.md | 4 +-
.../spark/launcher/CommandBuilderUtils.java | 2 +-
python/MANIFEST.in | 22 ++
python/README.md | 32 +++
python/pyspark/__init__.py | 1 +
python/pyspark/find_spark_home.py | 74 +++++++
python/pyspark/java_gateway.py | 3 +-
python/pyspark/version.py | 19 ++
python/setup.cfg | 22 ++
python/setup.py | 209 +++++++++++++++++++
31 files changed, 660 insertions(+), 24 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/a36a76ac/.gitignore
----------------------------------------------------------------------
diff --git a/.gitignore b/.gitignore
index 39d17e1..5634a43 100644
--- a/.gitignore
+++ b/.gitignore
@@ -57,6 +57,8 @@ project/plugins/project/build.properties
project/plugins/src_managed/
project/plugins/target/
python/lib/pyspark.zip
+python/deps
+python/pyspark/python
reports/
scalastyle-on-compile.generated.xml
scalastyle-output.xml
http://git-wip-us.apache.org/repos/asf/spark/blob/a36a76ac/bin/beeline
----------------------------------------------------------------------
diff --git a/bin/beeline b/bin/beeline
index 1627626..0585346 100755
--- a/bin/beeline
+++ b/bin/beeline
@@ -25,7 +25,7 @@ set -o posix
# Figure out if SPARK_HOME is set
if [ -z "${SPARK_HOME}" ]; then
- export SPARK_HOME="$(cd "`dirname "$0"`"/..; pwd)"
+ source "$(dirname "$0")"/find-spark-home
fi
CLASS="org.apache.hive.beeline.BeeLine"
http://git-wip-us.apache.org/repos/asf/spark/blob/a36a76ac/bin/find-spark-home
----------------------------------------------------------------------
diff --git a/bin/find-spark-home b/bin/find-spark-home
new file mode 100755
index 0000000..fa78407
--- /dev/null
+++ b/bin/find-spark-home
@@ -0,0 +1,41 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Attempts to find a proper value for SPARK_HOME. Should be included using "source" directive.
+
+FIND_SPARK_HOME_PYTHON_SCRIPT="$(cd "$(dirname "$0")"; pwd)/find_spark_home.py"
+
+# Short cirtuit if the user already has this set.
+if [ ! -z "${SPARK_HOME}" ]; then
+ exit 0
+elif [ ! -f "$FIND_SPARK_HOME_PYTHON_SCRIPT" ]; then
+ # If we are not in the same directory as find_spark_home.py we are not pip installed so we don't
+ # need to search the different Python directories for a Spark installation.
+ # Note only that, if the user has pip installed PySpark but is directly calling pyspark-shell or
+ # spark-submit in another directory we want to use that version of PySpark rather than the
+ # pip installed version of PySpark.
+ export SPARK_HOME="$(cd "$(dirname "$0")"/..; pwd)"
+else
+ # We are pip installed, use the Python script to resolve a reasonable SPARK_HOME
+ # Default to standard python interpreter unless told otherwise
+ if [[ -z "$PYSPARK_DRIVER_PYTHON" ]]; then
+ PYSPARK_DRIVER_PYTHON="${PYSPARK_PYTHON:-"python"}"
+ fi
+ export SPARK_HOME=$($PYSPARK_DRIVER_PYTHON "$FIND_SPARK_HOME_PYTHON_SCRIPT")
+fi
http://git-wip-us.apache.org/repos/asf/spark/blob/a36a76ac/bin/load-spark-env.sh
----------------------------------------------------------------------
diff --git a/bin/load-spark-env.sh b/bin/load-spark-env.sh
index eaea964..8a2f709 100644
--- a/bin/load-spark-env.sh
+++ b/bin/load-spark-env.sh
@@ -23,7 +23,7 @@
# Figure out where Spark is installed
if [ -z "${SPARK_HOME}" ]; then
- export SPARK_HOME="$(cd "`dirname "$0"`"/..; pwd)"
+ source "$(dirname "$0")"/find-spark-home
fi
if [ -z "$SPARK_ENV_LOADED" ]; then
http://git-wip-us.apache.org/repos/asf/spark/blob/a36a76ac/bin/pyspark
----------------------------------------------------------------------
diff --git a/bin/pyspark b/bin/pyspark
index d6b3ab0..98387c2 100755
--- a/bin/pyspark
+++ b/bin/pyspark
@@ -18,7 +18,7 @@
#
if [ -z "${SPARK_HOME}" ]; then
- export SPARK_HOME="$(cd "`dirname "$0"`"/..; pwd)"
+ source "$(dirname "$0")"/find-spark-home
fi
source "${SPARK_HOME}"/bin/load-spark-env.sh
@@ -46,7 +46,7 @@ WORKS_WITH_IPYTHON=$(python -c 'import sys; print(sys.version_info >= (2, 7, 0))
# Determine the Python executable to use for the executors:
if [[ -z "$PYSPARK_PYTHON" ]]; then
- if [[ $PYSPARK_DRIVER_PYTHON == *ipython* && ! WORKS_WITH_IPYTHON ]]; then
+ if [[ $PYSPARK_DRIVER_PYTHON == *ipython* && ! $WORKS_WITH_IPYTHON ]]; then
echo "IPython requires Python 2.7+; please install python2.7 or set PYSPARK_PYTHON" 1>&2
exit 1
else
@@ -68,7 +68,7 @@ if [[ -n "$SPARK_TESTING" ]]; then
unset YARN_CONF_DIR
unset HADOOP_CONF_DIR
export PYTHONHASHSEED=0
- exec "$PYSPARK_DRIVER_PYTHON" -m $1
+ exec "$PYSPARK_DRIVER_PYTHON" -m "$1"
exit
fi
http://git-wip-us.apache.org/repos/asf/spark/blob/a36a76ac/bin/run-example
----------------------------------------------------------------------
diff --git a/bin/run-example b/bin/run-example
index dd0e3c4..4ba5399 100755
--- a/bin/run-example
+++ b/bin/run-example
@@ -18,7 +18,7 @@
#
if [ -z "${SPARK_HOME}" ]; then
- export SPARK_HOME="$(cd "`dirname "$0"`"/..; pwd)"
+ source "$(dirname "$0")"/find-spark-home
fi
export _SPARK_CMD_USAGE="Usage: ./bin/run-example [options] example-class [example args]"
http://git-wip-us.apache.org/repos/asf/spark/blob/a36a76ac/bin/spark-class
----------------------------------------------------------------------
diff --git a/bin/spark-class b/bin/spark-class
index 377c8d1..77ea40c 100755
--- a/bin/spark-class
+++ b/bin/spark-class
@@ -18,7 +18,7 @@
#
if [ -z "${SPARK_HOME}" ]; then
- export SPARK_HOME="$(cd "`dirname "$0"`"/..; pwd)"
+ source "$(dirname "$0")"/find-spark-home
fi
. "${SPARK_HOME}"/bin/load-spark-env.sh
@@ -27,7 +27,7 @@ fi
if [ -n "${JAVA_HOME}" ]; then
RUNNER="${JAVA_HOME}/bin/java"
else
- if [ `command -v java` ]; then
+ if [ "$(command -v java)" ]; then
RUNNER="java"
else
echo "JAVA_HOME is not set" >&2
@@ -36,7 +36,7 @@ else
fi
# Find Spark jars.
-if [ -f "${SPARK_HOME}/RELEASE" ]; then
+if [ -d "${SPARK_HOME}/jars" ]; then
SPARK_JARS_DIR="${SPARK_HOME}/jars"
else
SPARK_JARS_DIR="${SPARK_HOME}/assembly/target/scala-$SPARK_SCALA_VERSION/jars"
http://git-wip-us.apache.org/repos/asf/spark/blob/a36a76ac/bin/spark-shell
----------------------------------------------------------------------
diff --git a/bin/spark-shell b/bin/spark-shell
index 6583b5b..421f36c 100755
--- a/bin/spark-shell
+++ b/bin/spark-shell
@@ -21,7 +21,7 @@
# Shell script for starting the Spark Shell REPL
cygwin=false
-case "`uname`" in
+case "$(uname)" in
CYGWIN*) cygwin=true;;
esac
@@ -29,7 +29,7 @@ esac
set -o posix
if [ -z "${SPARK_HOME}" ]; then
- export SPARK_HOME="$(cd "`dirname "$0"`"/..; pwd)"
+ source "$(dirname "$0")"/find-spark-home
fi
export _SPARK_CMD_USAGE="Usage: ./bin/spark-shell [options]"
http://git-wip-us.apache.org/repos/asf/spark/blob/a36a76ac/bin/spark-sql
----------------------------------------------------------------------
diff --git a/bin/spark-sql b/bin/spark-sql
index 970d12c..b08b944 100755
--- a/bin/spark-sql
+++ b/bin/spark-sql
@@ -18,7 +18,7 @@
#
if [ -z "${SPARK_HOME}" ]; then
- export SPARK_HOME="$(cd "`dirname "$0"`"/..; pwd)"
+ source "$(dirname "$0")"/find-spark-home
fi
export _SPARK_CMD_USAGE="Usage: ./bin/spark-sql [options] [cli option]"
http://git-wip-us.apache.org/repos/asf/spark/blob/a36a76ac/bin/spark-submit
----------------------------------------------------------------------
diff --git a/bin/spark-submit b/bin/spark-submit
index 023f9c1..4e9d361 100755
--- a/bin/spark-submit
+++ b/bin/spark-submit
@@ -18,7 +18,7 @@
#
if [ -z "${SPARK_HOME}" ]; then
- export SPARK_HOME="$(cd "`dirname "$0"`"/..; pwd)"
+ source "$(dirname "$0")"/find-spark-home
fi
# disable randomized hash for string in Python 3.3+
http://git-wip-us.apache.org/repos/asf/spark/blob/a36a76ac/bin/sparkR
----------------------------------------------------------------------
diff --git a/bin/sparkR b/bin/sparkR
index 2c07a82..29ab10d 100755
--- a/bin/sparkR
+++ b/bin/sparkR
@@ -18,7 +18,7 @@
#
if [ -z "${SPARK_HOME}" ]; then
- export SPARK_HOME="$(cd "`dirname "$0"`"/..; pwd)"
+ source "$(dirname "$0")"/find-spark-home
fi
source "${SPARK_HOME}"/bin/load-spark-env.sh
http://git-wip-us.apache.org/repos/asf/spark/blob/a36a76ac/dev/create-release/release-build.sh
----------------------------------------------------------------------
diff --git a/dev/create-release/release-build.sh b/dev/create-release/release-build.sh
index 81f0d63..1dbfa3b 100755
--- a/dev/create-release/release-build.sh
+++ b/dev/create-release/release-build.sh
@@ -162,14 +162,35 @@ if [[ "$1" == "package" ]]; then
export ZINC_PORT=$ZINC_PORT
echo "Creating distribution: $NAME ($FLAGS)"
+ # Write out the NAME and VERSION to PySpark version info we rewrite the - into a . and SNAPSHOT
+ # to dev0 to be closer to PEP440. We use the NAME as a "local version".
+ PYSPARK_VERSION=`echo "$SPARK_VERSION+$NAME" | sed -r "s/-/./" | sed -r "s/SNAPSHOT/dev0/"`
+ echo "__version__='$PYSPARK_VERSION'" > python/pyspark/version.py
+
# Get maven home set by MVN
MVN_HOME=`$MVN -version 2>&1 | grep 'Maven home' | awk '{print $NF}'`
- ./dev/make-distribution.sh --name $NAME --mvn $MVN_HOME/bin/mvn --tgz $FLAGS \
+ echo "Creating distribution"
+ ./dev/make-distribution.sh --name $NAME --mvn $MVN_HOME/bin/mvn --tgz --pip $FLAGS \
-DzincPort=$ZINC_PORT 2>&1 > ../binary-release-$NAME.log
cd ..
- cp spark-$SPARK_VERSION-bin-$NAME/spark-$SPARK_VERSION-bin-$NAME.tgz .
+ echo "Copying and signing python distribution"
+ PYTHON_DIST_NAME=pyspark-$PYSPARK_VERSION.tar.gz
+ cp spark-$SPARK_VERSION-bin-$NAME/python/dist/$PYTHON_DIST_NAME .
+
+ echo $GPG_PASSPHRASE | $GPG --passphrase-fd 0 --armour \
+ --output $PYTHON_DIST_NAME.asc \
+ --detach-sig $PYTHON_DIST_NAME
+ echo $GPG_PASSPHRASE | $GPG --passphrase-fd 0 --print-md \
+ MD5 $PYTHON_DIST_NAME > \
+ $PYTHON_DIST_NAME.md5
+ echo $GPG_PASSPHRASE | $GPG --passphrase-fd 0 --print-md \
+ SHA512 $PYTHON_DIST_NAME > \
+ $PYTHON_DIST_NAME.sha
+
+ echo "Copying and signing regular binary distribution"
+ cp spark-$SPARK_VERSION-bin-$NAME/spark-$SPARK_VERSION-bin-$NAME.tgz .
echo $GPG_PASSPHRASE | $GPG --passphrase-fd 0 --armour \
--output spark-$SPARK_VERSION-bin-$NAME.tgz.asc \
--detach-sig spark-$SPARK_VERSION-bin-$NAME.tgz
@@ -208,6 +229,7 @@ if [[ "$1" == "package" ]]; then
# Re-upload a second time and leave the files in the timestamped upload directory:
LFTP mkdir -p $dest_dir
LFTP mput -O $dest_dir 'spark-*'
+ LFTP mput -O $dest_dir 'pyspark-*'
exit 0
fi
http://git-wip-us.apache.org/repos/asf/spark/blob/a36a76ac/dev/create-release/release-tag.sh
----------------------------------------------------------------------
diff --git a/dev/create-release/release-tag.sh b/dev/create-release/release-tag.sh
index b7e5100..370a62c 100755
--- a/dev/create-release/release-tag.sh
+++ b/dev/create-release/release-tag.sh
@@ -65,6 +65,7 @@ sed -i".tmp1" 's/Version.*$/Version: '"$RELEASE_VERSION"'/g' R/pkg/DESCRIPTION
# Set the release version in docs
sed -i".tmp1" 's/SPARK_VERSION:.*$/SPARK_VERSION: '"$RELEASE_VERSION"'/g' docs/_config.yml
sed -i".tmp2" 's/SPARK_VERSION_SHORT:.*$/SPARK_VERSION_SHORT: '"$RELEASE_VERSION"'/g' docs/_config.yml
+sed -i".tmp3" 's/__version__ = .*$/__version__ = "'"$RELEASE_VERSION"'"/' python/pyspark/version.py
git commit -a -m "Preparing Spark release $RELEASE_TAG"
echo "Creating tag $RELEASE_TAG at the head of $GIT_BRANCH"
@@ -74,12 +75,16 @@ git tag $RELEASE_TAG
$MVN versions:set -DnewVersion=$NEXT_VERSION | grep -v "no value" # silence logs
# Remove -SNAPSHOT before setting the R version as R expects version strings to only have numbers
R_NEXT_VERSION=`echo $NEXT_VERSION | sed 's/-SNAPSHOT//g'`
-sed -i".tmp2" 's/Version.*$/Version: '"$R_NEXT_VERSION"'/g' R/pkg/DESCRIPTION
+sed -i".tmp4" 's/Version.*$/Version: '"$R_NEXT_VERSION"'/g' R/pkg/DESCRIPTION
+# Write out the R_NEXT_VERSION to PySpark version info we use dev0 instead of SNAPSHOT to be closer
+# to PEP440.
+sed -i".tmp5" 's/__version__ = .*$/__version__ = "'"$R_NEXT_VERSION.dev0"'"/' python/pyspark/version.py
+
# Update docs with next version
-sed -i".tmp3" 's/SPARK_VERSION:.*$/SPARK_VERSION: '"$NEXT_VERSION"'/g' docs/_config.yml
+sed -i".tmp6" 's/SPARK_VERSION:.*$/SPARK_VERSION: '"$NEXT_VERSION"'/g' docs/_config.yml
# Use R version for short version
-sed -i".tmp4" 's/SPARK_VERSION_SHORT:.*$/SPARK_VERSION_SHORT: '"$R_NEXT_VERSION"'/g' docs/_config.yml
+sed -i".tmp7" 's/SPARK_VERSION_SHORT:.*$/SPARK_VERSION_SHORT: '"$R_NEXT_VERSION"'/g' docs/_config.yml
git commit -a -m "Preparing development version $NEXT_VERSION"
http://git-wip-us.apache.org/repos/asf/spark/blob/a36a76ac/dev/lint-python
----------------------------------------------------------------------
diff --git a/dev/lint-python b/dev/lint-python
index 6348704..3f878c2 100755
--- a/dev/lint-python
+++ b/dev/lint-python
@@ -20,7 +20,9 @@
SCRIPT_DIR="$( cd "$( dirname "$0" )" && pwd )"
SPARK_ROOT_DIR="$(dirname "$SCRIPT_DIR")"
PATHS_TO_CHECK="./python/pyspark/ ./examples/src/main/python/ ./dev/sparktestsupport"
-PATHS_TO_CHECK="$PATHS_TO_CHECK ./dev/run-tests.py ./python/run-tests.py ./dev/run-tests-jenkins.py"
+# TODO: fix pep8 errors with the rest of the Python scripts under dev
+PATHS_TO_CHECK="$PATHS_TO_CHECK ./dev/run-tests.py ./python/*.py ./dev/run-tests-jenkins.py"
+PATHS_TO_CHECK="$PATHS_TO_CHECK ./dev/pip-sanity-check.py"
PEP8_REPORT_PATH="$SPARK_ROOT_DIR/dev/pep8-report.txt"
PYLINT_REPORT_PATH="$SPARK_ROOT_DIR/dev/pylint-report.txt"
PYLINT_INSTALL_INFO="$SPARK_ROOT_DIR/dev/pylint-info.txt"
http://git-wip-us.apache.org/repos/asf/spark/blob/a36a76ac/dev/make-distribution.sh
----------------------------------------------------------------------
diff --git a/dev/make-distribution.sh b/dev/make-distribution.sh
index 9be4fdf..49b46fb 100755
--- a/dev/make-distribution.sh
+++ b/dev/make-distribution.sh
@@ -33,6 +33,7 @@ SPARK_HOME="$(cd "`dirname "$0"`/.."; pwd)"
DISTDIR="$SPARK_HOME/dist"
MAKE_TGZ=false
+MAKE_PIP=false
NAME=none
MVN="$SPARK_HOME/build/mvn"
@@ -40,7 +41,7 @@ function exit_with_usage {
echo "make-distribution.sh - tool for making binary distributions of Spark"
echo ""
echo "usage:"
- cl_options="[--name] [--tgz] [--mvn <mvn-command>]"
+ cl_options="[--name] [--tgz] [--pip] [--mvn <mvn-command>]"
echo "make-distribution.sh $cl_options <maven build options>"
echo "See Spark's \"Building Spark\" doc for correct Maven options."
echo ""
@@ -67,6 +68,9 @@ while (( "$#" )); do
--tgz)
MAKE_TGZ=true
;;
+ --pip)
+ MAKE_PIP=true
+ ;;
--mvn)
MVN="$2"
shift
@@ -201,6 +205,16 @@ fi
# Copy data files
cp -r "$SPARK_HOME/data" "$DISTDIR"
+# Make pip package
+if [ "$MAKE_PIP" == "true" ]; then
+ echo "Building python distribution package"
+ cd $SPARK_HOME/python
+ python setup.py sdist
+ cd ..
+else
+ echo "Skipping creating pip installable PySpark"
+fi
+
# Copy other things
mkdir "$DISTDIR"/conf
cp "$SPARK_HOME"/conf/*.template "$DISTDIR"/conf
http://git-wip-us.apache.org/repos/asf/spark/blob/a36a76ac/dev/pip-sanity-check.py
----------------------------------------------------------------------
diff --git a/dev/pip-sanity-check.py b/dev/pip-sanity-check.py
new file mode 100644
index 0000000..430c2ab
--- /dev/null
+++ b/dev/pip-sanity-check.py
@@ -0,0 +1,36 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import print_function
+
+from pyspark.sql import SparkSession
+import sys
+
+if __name__ == "__main__":
+ spark = SparkSession\
+ .builder\
+ .appName("PipSanityCheck")\
+ .getOrCreate()
+ sc = spark.sparkContext
+ rdd = sc.parallelize(range(100), 10)
+ value = rdd.reduce(lambda x, y: x + y)
+ if (value != 4950):
+ print("Value {0} did not match expected value.".format(value), file=sys.stderr)
+ sys.exit(-1)
+ print("Successfully ran pip sanity check")
+
+ spark.stop()
http://git-wip-us.apache.org/repos/asf/spark/blob/a36a76ac/dev/run-pip-tests
----------------------------------------------------------------------
diff --git a/dev/run-pip-tests b/dev/run-pip-tests
new file mode 100755
index 0000000..e1da18e
--- /dev/null
+++ b/dev/run-pip-tests
@@ -0,0 +1,115 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Stop on error
+set -e
+# Set nullglob for when we are checking existence based on globs
+shopt -s nullglob
+
+FWDIR="$(cd "$(dirname "$0")"/..; pwd)"
+cd "$FWDIR"
+
+echo "Constucting virtual env for testing"
+VIRTUALENV_BASE=$(mktemp -d)
+
+# Clean up the virtual env enviroment used if we created one.
+function delete_virtualenv() {
+ echo "Cleaning up temporary directory - $VIRTUALENV_BASE"
+ rm -rf "$VIRTUALENV_BASE"
+}
+trap delete_virtualenv EXIT
+
+# Some systems don't have pip or virtualenv - in those cases our tests won't work.
+if ! hash virtualenv 2>/dev/null; then
+ echo "Missing virtualenv skipping pip installability tests."
+ exit 0
+fi
+if ! hash pip 2>/dev/null; then
+ echo "Missing pip, skipping pip installability tests."
+ exit 0
+fi
+
+# Figure out which Python execs we should test pip installation with
+PYTHON_EXECS=()
+if hash python2 2>/dev/null; then
+ # We do this since we are testing with virtualenv and the default virtual env python
+ # is in /usr/bin/python
+ PYTHON_EXECS+=('python2')
+elif hash python 2>/dev/null; then
+ # If python2 isn't installed fallback to python if available
+ PYTHON_EXECS+=('python')
+fi
+if hash python3 2>/dev/null; then
+ PYTHON_EXECS+=('python3')
+fi
+
+# Determine which version of PySpark we are building for archive name
+PYSPARK_VERSION=$(python -c "exec(open('python/pyspark/version.py').read());print __version__")
+PYSPARK_DIST="$FWDIR/python/dist/pyspark-$PYSPARK_VERSION.tar.gz"
+# The pip install options we use for all the pip commands
+PIP_OPTIONS="--upgrade --no-cache-dir --force-reinstall "
+# Test both regular user and edit/dev install modes.
+PIP_COMMANDS=("pip install $PIP_OPTIONS $PYSPARK_DIST"
+ "pip install $PIP_OPTIONS -e python/")
+
+for python in "${PYTHON_EXECS[@]}"; do
+ for install_command in "${PIP_COMMANDS[@]}"; do
+ echo "Testing pip installation with python $python"
+ # Create a temp directory for us to work in and save its name to a file for cleanup
+ echo "Using $VIRTUALENV_BASE for virtualenv"
+ VIRTUALENV_PATH="$VIRTUALENV_BASE"/$python
+ rm -rf "$VIRTUALENV_PATH"
+ mkdir -p "$VIRTUALENV_PATH"
+ virtualenv --python=$python "$VIRTUALENV_PATH"
+ source "$VIRTUALENV_PATH"/bin/activate
+ # Upgrade pip
+ pip install --upgrade pip
+
+ echo "Creating pip installable source dist"
+ cd "$FWDIR"/python
+ $python setup.py sdist
+
+
+ echo "Installing dist into virtual env"
+ cd dist
+ # Verify that the dist directory only contains one thing to install
+ sdists=(*.tar.gz)
+ if [ ${#sdists[@]} -ne 1 ]; then
+ echo "Unexpected number of targets found in dist directory - please cleanup existing sdists first."
+ exit -1
+ fi
+ # Do the actual installation
+ cd "$FWDIR"
+ $install_command
+
+ cd /
+
+ echo "Run basic sanity check on pip installed version with spark-submit"
+ spark-submit "$FWDIR"/dev/pip-sanity-check.py
+ echo "Run basic sanity check with import based"
+ python "$FWDIR"/dev/pip-sanity-check.py
+ echo "Run the tests for context.py"
+ python "$FWDIR"/python/pyspark/context.py
+
+ cd "$FWDIR"
+
+ done
+done
+
+exit 0
http://git-wip-us.apache.org/repos/asf/spark/blob/a36a76ac/dev/run-tests-jenkins.py
----------------------------------------------------------------------
diff --git a/dev/run-tests-jenkins.py b/dev/run-tests-jenkins.py
index a48d918..1d1e72f 100755
--- a/dev/run-tests-jenkins.py
+++ b/dev/run-tests-jenkins.py
@@ -128,6 +128,7 @@ def run_tests(tests_timeout):
ERROR_CODES["BLOCK_MIMA"]: 'MiMa tests',
ERROR_CODES["BLOCK_SPARK_UNIT_TESTS"]: 'Spark unit tests',
ERROR_CODES["BLOCK_PYSPARK_UNIT_TESTS"]: 'PySpark unit tests',
+ ERROR_CODES["BLOCK_PYSPARK_PIP_TESTS"]: 'PySpark pip packaging tests',
ERROR_CODES["BLOCK_SPARKR_UNIT_TESTS"]: 'SparkR unit tests',
ERROR_CODES["BLOCK_TIMEOUT"]: 'from timeout after a configured wait of \`%s\`' % (
tests_timeout)
http://git-wip-us.apache.org/repos/asf/spark/blob/a36a76ac/dev/run-tests.py
----------------------------------------------------------------------
diff --git a/dev/run-tests.py b/dev/run-tests.py
index 5d661f5..ab285ac 100755
--- a/dev/run-tests.py
+++ b/dev/run-tests.py
@@ -432,6 +432,12 @@ def run_python_tests(test_modules, parallelism):
run_cmd(command)
+def run_python_packaging_tests():
+ set_title_and_block("Running PySpark packaging tests", "BLOCK_PYSPARK_PIP_TESTS")
+ command = [os.path.join(SPARK_HOME, "dev", "run-pip-tests")]
+ run_cmd(command)
+
+
def run_build_tests():
set_title_and_block("Running build tests", "BLOCK_BUILD_TESTS")
run_cmd([os.path.join(SPARK_HOME, "dev", "test-dependencies.sh")])
@@ -583,6 +589,7 @@ def main():
modules_with_python_tests = [m for m in test_modules if m.python_test_goals]
if modules_with_python_tests:
run_python_tests(modules_with_python_tests, opts.parallelism)
+ run_python_packaging_tests()
if any(m.should_run_r_tests for m in test_modules):
run_sparkr_tests()
http://git-wip-us.apache.org/repos/asf/spark/blob/a36a76ac/dev/sparktestsupport/__init__.py
----------------------------------------------------------------------
diff --git a/dev/sparktestsupport/__init__.py b/dev/sparktestsupport/__init__.py
index 89015f8..38f25da 100644
--- a/dev/sparktestsupport/__init__.py
+++ b/dev/sparktestsupport/__init__.py
@@ -33,5 +33,6 @@ ERROR_CODES = {
"BLOCK_SPARKR_UNIT_TESTS": 20,
"BLOCK_JAVA_STYLE": 21,
"BLOCK_BUILD_TESTS": 22,
+ "BLOCK_PYSPARK_PIP_TESTS": 23,
"BLOCK_TIMEOUT": 124
}
http://git-wip-us.apache.org/repos/asf/spark/blob/a36a76ac/docs/building-spark.md
----------------------------------------------------------------------
diff --git a/docs/building-spark.md b/docs/building-spark.md
index 2b404bd..88da0cc 100644
--- a/docs/building-spark.md
+++ b/docs/building-spark.md
@@ -265,6 +265,14 @@ or
Java 8 tests are automatically enabled when a Java 8 JDK is detected.
If you have JDK 8 installed but it is not the system default, you can set JAVA_HOME to point to JDK 8 before running the tests.
+## PySpark pip installable
+
+If you are building Spark for use in a Python environment and you wish to pip install it, you will first need to build the Spark JARs as described above. Then you can construct an sdist package suitable for setup.py and pip installable package.
+
+ cd python; python setup.py sdist
+
+**Note:** Due to packaging requirements you can not directly pip install from the Python directory, rather you must first build the sdist package as described above.
+
## PySpark Tests with Maven
If you are building PySpark and wish to run the PySpark tests you will need to build Spark with Hive support.
http://git-wip-us.apache.org/repos/asf/spark/blob/a36a76ac/docs/index.md
----------------------------------------------------------------------
diff --git a/docs/index.md b/docs/index.md
index fe51439..39de11d 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -14,7 +14,9 @@ It also supports a rich set of higher-level tools including [Spark SQL](sql-prog
Get Spark from the [downloads page](http://spark.apache.org/downloads.html) of the project website. This documentation is for Spark version {{site.SPARK_VERSION}}. Spark uses Hadoop's client libraries for HDFS and YARN. Downloads are pre-packaged for a handful of popular Hadoop versions.
Users can also download a "Hadoop free" binary and run Spark with any Hadoop version
-[by augmenting Spark's classpath](hadoop-provided.html).
+[by augmenting Spark's classpath](hadoop-provided.html).
+Scala and Java users can include Spark in their projects using its maven cooridnates and in the future Python users can also install Spark from PyPI.
+
If you'd like to build Spark from
source, visit [Building Spark](building-spark.html).
http://git-wip-us.apache.org/repos/asf/spark/blob/a36a76ac/launcher/src/main/java/org/apache/spark/launcher/CommandBuilderUtils.java
----------------------------------------------------------------------
diff --git a/launcher/src/main/java/org/apache/spark/launcher/CommandBuilderUtils.java b/launcher/src/main/java/org/apache/spark/launcher/CommandBuilderUtils.java
index 62a2200..250b2a8 100644
--- a/launcher/src/main/java/org/apache/spark/launcher/CommandBuilderUtils.java
+++ b/launcher/src/main/java/org/apache/spark/launcher/CommandBuilderUtils.java
@@ -357,7 +357,7 @@ class CommandBuilderUtils {
static String findJarsDir(String sparkHome, String scalaVersion, boolean failIfNotFound) {
// TODO: change to the correct directory once the assembly build is changed.
File libdir;
- if (new File(sparkHome, "RELEASE").isFile()) {
+ if (new File(sparkHome, "jars").isDirectory()) {
libdir = new File(sparkHome, "jars");
checkState(!failIfNotFound || libdir.isDirectory(),
"Library directory '%s' does not exist.",
http://git-wip-us.apache.org/repos/asf/spark/blob/a36a76ac/python/MANIFEST.in
----------------------------------------------------------------------
diff --git a/python/MANIFEST.in b/python/MANIFEST.in
new file mode 100644
index 0000000..bbcce1b
--- /dev/null
+++ b/python/MANIFEST.in
@@ -0,0 +1,22 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+global-exclude *.py[cod] __pycache__ .DS_Store
+recursive-include deps/jars *.jar
+graft deps/bin
+recursive-include deps/examples *.py
+recursive-include lib *.zip
+include README.md
http://git-wip-us.apache.org/repos/asf/spark/blob/a36a76ac/python/README.md
----------------------------------------------------------------------
diff --git a/python/README.md b/python/README.md
new file mode 100644
index 0000000..0a5c801
--- /dev/null
+++ b/python/README.md
@@ -0,0 +1,32 @@
+# Apache Spark
+
+Spark is a fast and general cluster computing system for Big Data. It provides
+high-level APIs in Scala, Java, Python, and R, and an optimized engine that
+supports general computation graphs for data analysis. It also supports a
+rich set of higher-level tools including Spark SQL for SQL and DataFrames,
+MLlib for machine learning, GraphX for graph processing,
+and Spark Streaming for stream processing.
+
+<http://spark.apache.org/>
+
+## Online Documentation
+
+You can find the latest Spark documentation, including a programming
+guide, on the [project web page](http://spark.apache.org/documentation.html)
+
+
+## Python Packaging
+
+This README file only contains basic information related to pip installed PySpark.
+This packaging is currently experimental and may change in future versions (although we will do our best to keep compatibility).
+Using PySpark requires the Spark JARs, and if you are building this from source please see the builder instructions at
+["Building Spark"](http://spark.apache.org/docs/latest/building-spark.html).
+
+The Python packaging for Spark is not intended to replace all of the other use cases. This Python packaged version of Spark is suitable for interacting with an existing cluster (be it Spark standalone, YARN, or Mesos) - but does not contain the tools required to setup your own standalone Spark cluster. You can download the full version of Spark from the [Apache Spark downloads page](http://spark.apache.org/downloads.html).
+
+
+**NOTE:** If you are using this with a Spark standalone cluster you must ensure that the version (including minor version) matches or you may experience odd errors.
+
+## Python Requirements
+
+At its core PySpark depends on Py4J (currently version 0.10.4), but additional sub-packages have their own requirements (including numpy and pandas).
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/spark/blob/a36a76ac/python/pyspark/__init__.py
----------------------------------------------------------------------
diff --git a/python/pyspark/__init__.py b/python/pyspark/__init__.py
index ec16874..5f93586 100644
--- a/python/pyspark/__init__.py
+++ b/python/pyspark/__init__.py
@@ -50,6 +50,7 @@ from pyspark.broadcast import Broadcast
from pyspark.serializers import MarshalSerializer, PickleSerializer
from pyspark.status import *
from pyspark.profiler import Profiler, BasicProfiler
+from pyspark.version import __version__
def since(version):
http://git-wip-us.apache.org/repos/asf/spark/blob/a36a76ac/python/pyspark/find_spark_home.py
----------------------------------------------------------------------
diff --git a/python/pyspark/find_spark_home.py b/python/pyspark/find_spark_home.py
new file mode 100755
index 0000000..212a618
--- /dev/null
+++ b/python/pyspark/find_spark_home.py
@@ -0,0 +1,74 @@
+#!/usr/bin/env python
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This script attempt to determine the correct setting for SPARK_HOME given
+# that Spark may have been installed on the system with pip.
+
+from __future__ import print_function
+import os
+import sys
+
+
+def _find_spark_home():
+ """Find the SPARK_HOME."""
+ # If the enviroment has SPARK_HOME set trust it.
+ if "SPARK_HOME" in os.environ:
+ return os.environ["SPARK_HOME"]
+
+ def is_spark_home(path):
+ """Takes a path and returns true if the provided path could be a reasonable SPARK_HOME"""
+ return (os.path.isfile(os.path.join(path, "bin/spark-submit")) and
+ (os.path.isdir(os.path.join(path, "jars")) or
+ os.path.isdir(os.path.join(path, "assembly"))))
+
+ paths = ["../", os.path.dirname(os.path.realpath(__file__))]
+
+ # Add the path of the PySpark module if it exists
+ if sys.version < "3":
+ import imp
+ try:
+ module_home = imp.find_module("pyspark")[1]
+ paths.append(module_home)
+ # If we are installed in edit mode also look two dirs up
+ paths.append(os.path.join(module_home, "../../"))
+ except ImportError:
+ # Not pip installed no worries
+ pass
+ else:
+ from importlib.util import find_spec
+ try:
+ module_home = os.path.dirname(find_spec("pyspark").origin)
+ paths.append(module_home)
+ # If we are installed in edit mode also look two dirs up
+ paths.append(os.path.join(module_home, "../../"))
+ except ImportError:
+ # Not pip installed no worries
+ pass
+
+ # Normalize the paths
+ paths = [os.path.abspath(p) for p in paths]
+
+ try:
+ return next(path for path in paths if is_spark_home(path))
+ except StopIteration:
+ print("Could not find valid SPARK_HOME while searching {0}".format(paths), file=sys.stderr)
+ exit(-1)
+
+if __name__ == "__main__":
+ print(_find_spark_home())
http://git-wip-us.apache.org/repos/asf/spark/blob/a36a76ac/python/pyspark/java_gateway.py
----------------------------------------------------------------------
diff --git a/python/pyspark/java_gateway.py b/python/pyspark/java_gateway.py
index c1cf843..3c783ae 100644
--- a/python/pyspark/java_gateway.py
+++ b/python/pyspark/java_gateway.py
@@ -29,6 +29,7 @@ if sys.version >= '3':
xrange = range
from py4j.java_gateway import java_import, JavaGateway, GatewayClient
+from pyspark.find_spark_home import _find_spark_home
from pyspark.serializers import read_int
@@ -41,7 +42,7 @@ def launch_gateway(conf=None):
if "PYSPARK_GATEWAY_PORT" in os.environ:
gateway_port = int(os.environ["PYSPARK_GATEWAY_PORT"])
else:
- SPARK_HOME = os.environ["SPARK_HOME"]
+ SPARK_HOME = _find_spark_home()
# Launch the Py4j gateway using Spark's run command so that we pick up the
# proper classpath and settings from spark-env.sh
on_windows = platform.system() == "Windows"
http://git-wip-us.apache.org/repos/asf/spark/blob/a36a76ac/python/pyspark/version.py
----------------------------------------------------------------------
diff --git a/python/pyspark/version.py b/python/pyspark/version.py
new file mode 100644
index 0000000..08a3016
--- /dev/null
+++ b/python/pyspark/version.py
@@ -0,0 +1,19 @@
+#!/usr/bin/env python
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+__version__ = "2.1.0.dev0"
http://git-wip-us.apache.org/repos/asf/spark/blob/a36a76ac/python/setup.cfg
----------------------------------------------------------------------
diff --git a/python/setup.cfg b/python/setup.cfg
new file mode 100644
index 0000000..d100b93
--- /dev/null
+++ b/python/setup.cfg
@@ -0,0 +1,22 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+[bdist_wheel]
+universal = 1
+
+[metadata]
+description-file = README.md
http://git-wip-us.apache.org/repos/asf/spark/blob/a36a76ac/python/setup.py
----------------------------------------------------------------------
diff --git a/python/setup.py b/python/setup.py
new file mode 100644
index 0000000..625aea0
--- /dev/null
+++ b/python/setup.py
@@ -0,0 +1,209 @@
+#!/usr/bin/env python
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import print_function
+import glob
+import os
+import sys
+from setuptools import setup, find_packages
+from shutil import copyfile, copytree, rmtree
+
+if sys.version_info < (2, 7):
+ print("Python versions prior to 2.7 are not supported for pip installed PySpark.",
+ file=sys.stderr)
+ exit(-1)
+
+try:
+ exec(open('pyspark/version.py').read())
+except IOError:
+ print("Failed to load PySpark version file for packaging. You must be in Spark's python dir.",
+ file=sys.stderr)
+ sys.exit(-1)
+VERSION = __version__
+# A temporary path so we can access above the Python project root and fetch scripts and jars we need
+TEMP_PATH = "deps"
+SPARK_HOME = os.path.abspath("../")
+
+# Provide guidance about how to use setup.py
+incorrect_invocation_message = """
+If you are installing pyspark from spark source, you must first build Spark and
+run sdist.
+
+ To build Spark with maven you can run:
+ ./build/mvn -DskipTests clean package
+ Building the source dist is done in the Python directory:
+ cd python
+ python setup.py sdist
+ pip install dist/*.tar.gz"""
+
+# Figure out where the jars are we need to package with PySpark.
+JARS_PATH = glob.glob(os.path.join(SPARK_HOME, "assembly/target/scala-*/jars/"))
+
+if len(JARS_PATH) == 1:
+ JARS_PATH = JARS_PATH[0]
+elif (os.path.isfile("../RELEASE") and len(glob.glob("../jars/spark*core*.jar")) == 1):
+ # Release mode puts the jars in a jars directory
+ JARS_PATH = os.path.join(SPARK_HOME, "jars")
+elif len(JARS_PATH) > 1:
+ print("Assembly jars exist for multiple scalas ({0}), please cleanup assembly/target".format(
+ JARS_PATH), file=sys.stderr)
+ sys.exit(-1)
+elif len(JARS_PATH) == 0 and not os.path.exists(TEMP_PATH):
+ print(incorrect_invocation_message, file=sys.stderr)
+ sys.exit(-1)
+
+EXAMPLES_PATH = os.path.join(SPARK_HOME, "examples/src/main/python")
+SCRIPTS_PATH = os.path.join(SPARK_HOME, "bin")
+SCRIPTS_TARGET = os.path.join(TEMP_PATH, "bin")
+JARS_TARGET = os.path.join(TEMP_PATH, "jars")
+EXAMPLES_TARGET = os.path.join(TEMP_PATH, "examples")
+
+
+# Check and see if we are under the spark path in which case we need to build the symlink farm.
+# This is important because we only want to build the symlink farm while under Spark otherwise we
+# want to use the symlink farm. And if the symlink farm exists under while under Spark (e.g. a
+# partially built sdist) we should error and have the user sort it out.
+in_spark = (os.path.isfile("../core/src/main/scala/org/apache/spark/SparkContext.scala") or
+ (os.path.isfile("../RELEASE") and len(glob.glob("../jars/spark*core*.jar")) == 1))
+
+
+def _supports_symlinks():
+ """Check if the system supports symlinks (e.g. *nix) or not."""
+ return getattr(os, "symlink", None) is not None
+
+
+if (in_spark):
+ # Construct links for setup
+ try:
+ os.mkdir(TEMP_PATH)
+ except:
+ print("Temp path for symlink to parent already exists {0}".format(TEMP_PATH),
+ file=sys.stderr)
+ exit(-1)
+
+try:
+ # We copy the shell script to be under pyspark/python/pyspark so that the launcher scripts
+ # find it where expected. The rest of the files aren't copied because they are accessed
+ # using Python imports instead which will be resolved correctly.
+ try:
+ os.makedirs("pyspark/python/pyspark")
+ except OSError:
+ # Don't worry if the directory already exists.
+ pass
+ copyfile("pyspark/shell.py", "pyspark/python/pyspark/shell.py")
+
+ if (in_spark):
+ # Construct the symlink farm - this is necessary since we can't refer to the path above the
+ # package root and we need to copy the jars and scripts which are up above the python root.
+ if _supports_symlinks():
+ os.symlink(JARS_PATH, JARS_TARGET)
+ os.symlink(SCRIPTS_PATH, SCRIPTS_TARGET)
+ os.symlink(EXAMPLES_PATH, EXAMPLES_TARGET)
+ else:
+ # For windows fall back to the slower copytree
+ copytree(JARS_PATH, JARS_TARGET)
+ copytree(SCRIPTS_PATH, SCRIPTS_TARGET)
+ copytree(EXAMPLES_PATH, EXAMPLES_TARGET)
+ else:
+ # If we are not inside of SPARK_HOME verify we have the required symlink farm
+ if not os.path.exists(JARS_TARGET):
+ print("To build packaging must be in the python directory under the SPARK_HOME.",
+ file=sys.stderr)
+
+ if not os.path.isdir(SCRIPTS_TARGET):
+ print(incorrect_invocation_message, file=sys.stderr)
+ exit(-1)
+
+ # Scripts directive requires a list of each script path and does not take wild cards.
+ script_names = os.listdir(SCRIPTS_TARGET)
+ scripts = list(map(lambda script: os.path.join(SCRIPTS_TARGET, script), script_names))
+ # We add find_spark_home.py to the bin directory we install so that pip installed PySpark
+ # will search for SPARK_HOME with Python.
+ scripts.append("pyspark/find_spark_home.py")
+
+ # Parse the README markdown file into rst for PyPI
+ long_description = "!!!!! missing pandoc do not upload to PyPI !!!!"
+ try:
+ import pypandoc
+ long_description = pypandoc.convert('README.md', 'rst')
+ except ImportError:
+ print("Could not import pypandoc - required to package PySpark", file=sys.stderr)
+
+ setup(
+ name='pyspark',
+ version=VERSION,
+ description='Apache Spark Python API',
+ long_description=long_description,
+ author='Spark Developers',
+ author_email='dev@spark.apache.org',
+ url='https://github.com/apache/spark/tree/master/python',
+ packages=['pyspark',
+ 'pyspark.mllib',
+ 'pyspark.ml',
+ 'pyspark.sql',
+ 'pyspark.streaming',
+ 'pyspark.bin',
+ 'pyspark.jars',
+ 'pyspark.python.pyspark',
+ 'pyspark.python.lib',
+ 'pyspark.examples.src.main.python'],
+ include_package_data=True,
+ package_dir={
+ 'pyspark.jars': 'deps/jars',
+ 'pyspark.bin': 'deps/bin',
+ 'pyspark.python.lib': 'lib',
+ 'pyspark.examples.src.main.python': 'deps/examples',
+ },
+ package_data={
+ 'pyspark.jars': ['*.jar'],
+ 'pyspark.bin': ['*'],
+ 'pyspark.python.lib': ['*.zip'],
+ 'pyspark.examples.src.main.python': ['*.py', '*/*.py']},
+ scripts=scripts,
+ license='http://www.apache.org/licenses/LICENSE-2.0',
+ install_requires=['py4j==0.10.4'],
+ setup_requires=['pypandoc'],
+ extras_require={
+ 'ml': ['numpy>=1.7'],
+ 'mllib': ['numpy>=1.7'],
+ 'sql': ['pandas']
+ },
+ classifiers=[
+ 'Development Status :: 5 - Production/Stable',
+ 'License :: OSI Approved :: Apache Software License',
+ 'Programming Language :: Python :: 2.7',
+ 'Programming Language :: Python :: 3',
+ 'Programming Language :: Python :: 3.4',
+ 'Programming Language :: Python :: 3.5',
+ 'Programming Language :: Python :: Implementation :: CPython',
+ 'Programming Language :: Python :: Implementation :: PyPy']
+ )
+finally:
+ # We only cleanup the symlink farm if we were in Spark, otherwise we are installing rather than
+ # packaging.
+ if (in_spark):
+ # Depending on cleaning up the symlink farm or copied version
+ if _supports_symlinks():
+ os.remove(os.path.join(TEMP_PATH, "jars"))
+ os.remove(os.path.join(TEMP_PATH, "bin"))
+ os.remove(os.path.join(TEMP_PATH, "examples"))
+ else:
+ rmtree(os.path.join(TEMP_PATH, "jars"))
+ rmtree(os.path.join(TEMP_PATH, "bin"))
+ rmtree(os.path.join(TEMP_PATH, "examples"))
+ os.rmdir(TEMP_PATH)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org