You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by jsnowacki <gi...@git.apache.org> on 2017/11/23 19:32:29 UTC
[GitHub] spark pull request #19807: [SPARK-22495] Fix setup of SPARK_HOME variable on...
GitHub user jsnowacki opened a pull request:
https://github.com/apache/spark/pull/19807
[SPARK-22495] Fix setup of SPARK_HOME variable on Windows
## What changes were proposed in this pull request?
This is a cherry pick of the original PR 19370 onto branch-2.2 as suggested in https://github.com/apache/spark/pull/19370#issuecomment-346526920.
Fixing the way how `SPARK_HOME` is resolved on Windows. While the previous version was working with the built release download, the set of directories changed slightly for the PySpark `pip` or `conda` install. This has been reflected in Linux files in `bin` but not for Windows `cmd` files.
First fix improves the way how the `jars` directory is found, as this was stoping Windows version of `pip/conda` install from working; JARs were not found by on Session/Context setup.
Second fix is adding `find-spark-home.cmd` script, which uses `find_spark_home.py` script, as the Linux version, to resolve `SPARK_HOME`. It is based on `find-spark-home` bash script, though, some operations are done in different order due to the `cmd` script language limitations. If environment variable is set, the Python script `find_spark_home.py` will not be run. The process can fail if Python is not installed, but it will mostly use this way if PySpark is installed via `pip/conda`, thus, there is some Python in the system.
## How was this patch tested?
Tested on local installation.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jsnowacki/spark-1 fix_spark_cmds_2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19807.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19807
----
commit bd24e470227437a52b07d95335579f1afcdda905
Author: Jakub Nowacki <j....@gmail.com>
Date: 2017-10-06T12:06:15Z
[SPARK-22495] Fix setup of SPARK_HOME variable on Windows
(cherry picked from commit b58f740)
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19807: [SPARK-22495] Fix setup of SPARK_HOME variable on...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/19807#discussion_r152889444
--- Diff: bin/find-spark-home.cmd ---
@@ -0,0 +1,60 @@
+@echo off
+
+rem
+rem Licensed to the Apache Software Foundation (ASF) under one or more
+rem contributor license agreements. See the NOTICE file distributed with
+rem this work for additional information regarding copyright ownership.
+rem The ASF licenses this file to You under the Apache License, Version 2.0
+rem (the "License"); you may not use this file except in compliance with
+rem the License. You may obtain a copy of the License at
+rem
+rem http://www.apache.org/licenses/LICENSE-2.0
+rem
+rem Unless required by applicable law or agreed to in writing, software
+rem distributed under the License is distributed on an "AS IS" BASIS,
+rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+rem See the License for the specific language governing permissions and
+rem limitations under the License.
+rem
+
+rem Path to Python script finding SPARK_HOME
+set FIND_SPARK_HOME_PYTHON_SCRIPT=%~dp0find_spark_home.py
+
+rem Default to standard python interpreter unless told otherwise
+set PYTHON_RUNNER=python
+rem If PYSPARK_DRIVER_PYTHON is set, it overwrites the python version
+if not "x%PYSPARK_DRIVER_PYTHON%"=="x" (
+ set PYTHON_RUNNER=%PYSPARK_DRIVER_PYTHON%
+)
+rem If PYSPARK_PYTHON is set, it overwrites the python version
+if not "x%PYSPARK_PYTHON%"=="x" (
+ set PYTHON_RUNNER=%PYSPARK_PYTHON%
+)
+
+rem If there is python installed, trying to use the root dir as SPARK_HOME
+where %PYTHON_RUNNER% > nul 2>$1
--- End diff --
There seems a typo here actually !! `where %PYTHON_RUNNER% > nul 2>$1` -> `where %PYTHON_RUNNER% > nul 2>&1`.
Will fix it up in master by myself soon.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19807: [SPARK-22495] Fix setup of SPARK_HOME variable on Window...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19807
**[Test build #84142 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84142/testReport)** for PR 19807 at commit [`bd24e47`](https://github.com/apache/spark/commit/bd24e470227437a52b07d95335579f1afcdda905).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19807: [SPARK-22495] Fix setup of SPARK_HOME variable on Window...
Posted by jsnowacki <gi...@git.apache.org>.
Github user jsnowacki commented on the issue:
https://github.com/apache/spark/pull/19807
@HyukjinKwon I've created this cherry picked PR onto branch-2.2. Please take a look if this is what you had in mind.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19807: [SPARK-22495] Fix setup of SPARK_HOME variable on...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/19807#discussion_r153021890
--- Diff: bin/find-spark-home.cmd ---
@@ -0,0 +1,60 @@
+@echo off
+
+rem
+rem Licensed to the Apache Software Foundation (ASF) under one or more
+rem contributor license agreements. See the NOTICE file distributed with
+rem this work for additional information regarding copyright ownership.
+rem The ASF licenses this file to You under the Apache License, Version 2.0
+rem (the "License"); you may not use this file except in compliance with
+rem the License. You may obtain a copy of the License at
+rem
+rem http://www.apache.org/licenses/LICENSE-2.0
+rem
+rem Unless required by applicable law or agreed to in writing, software
+rem distributed under the License is distributed on an "AS IS" BASIS,
+rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+rem See the License for the specific language governing permissions and
+rem limitations under the License.
+rem
+
+rem Path to Python script finding SPARK_HOME
+set FIND_SPARK_HOME_PYTHON_SCRIPT=%~dp0find_spark_home.py
+
+rem Default to standard python interpreter unless told otherwise
+set PYTHON_RUNNER=python
+rem If PYSPARK_DRIVER_PYTHON is set, it overwrites the python version
+if not "x%PYSPARK_DRIVER_PYTHON%"=="x" (
+ set PYTHON_RUNNER=%PYSPARK_DRIVER_PYTHON%
+)
+rem If PYSPARK_PYTHON is set, it overwrites the python version
+if not "x%PYSPARK_PYTHON%"=="x" (
+ set PYTHON_RUNNER=%PYSPARK_PYTHON%
+)
+
+rem If there is python installed, trying to use the root dir as SPARK_HOME
+where %PYTHON_RUNNER% > nul 2>$1
--- End diff --
oh yes
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19807: [SPARK-22495] Fix setup of SPARK_HOME variable on...
Posted by jsnowacki <gi...@git.apache.org>.
Github user jsnowacki closed the pull request at:
https://github.com/apache/spark/pull/19807
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19807: [SPARK-22495] Fix setup of SPARK_HOME variable on Window...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19807
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19807: [SPARK-22495] Fix setup of SPARK_HOME variable on Window...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/19807
I fixed it during merge and merged this to 2.2.
@jsnowacki thanks and please close this PR.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19807: [SPARK-22495] Fix setup of SPARK_HOME variable on Window...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19807
LGTM pending AppVeyor tests.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19807: [SPARK-22495] Fix setup of SPARK_HOME variable on Window...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19807
**[Test build #84142 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84142/testReport)** for PR 19807 at commit [`bd24e47`](https://github.com/apache/spark/commit/bd24e470227437a52b07d95335579f1afcdda905).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19807: [SPARK-22495] Fix setup of SPARK_HOME variable on Window...
Posted by jsnowacki <gi...@git.apache.org>.
Github user jsnowacki commented on the issue:
https://github.com/apache/spark/pull/19807
Thanks! Closing.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19807: [SPARK-22495] Fix setup of SPARK_HOME variable on Window...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19807
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84142/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19807: [SPARK-22495] Fix setup of SPARK_HOME variable on Window...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19807
Build started: [SparkR] `ALL` [](https://ci.appveyor.com/project/spark-test/spark/branch/4A7B521C-83BF-4F47-84AF-94D49BCBF40E)
Diff: https://github.com/apache/spark/compare/branch-2.2...spark-test:4A7B521C-83BF-4F47-84AF-94D49BCBF40E
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org