You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by bersprockets <gi...@git.apache.org> on 2018/06/25 02:16:46 UTC
[GitHub] spark pull request #21628: [SPARK-23776][DOC] Update instructions for runnin...
GitHub user bersprockets opened a pull request:
https://github.com/apache/spark/pull/21628
[SPARK-23776][DOC] Update instructions for running PySpark after building with SBT
## What changes were proposed in this pull request?
This update tells the reader how to build Spark with SBT such that pyspark-sql tests will succeed.
If you follow the current instructions for building Spark with SBT, pyspark/sql/udf.py fails with:
<pre>
AnalysisException: u'Can not load class test.org.apache.spark.sql.JavaStringLength, please make sure it is on the classpath;'
</pre>
## How was this patch tested?
I ran the doc build command (SKIP_API=1 jekyll build) and eyeballed the result.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/bersprockets/spark SPARK-23776_doc
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21628.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21628
----
commit 9fcd05d7cb52a68bea930625605013397b4989f6
Author: Bruce Robbins <be...@...>
Date: 2018-06-25T02:07:12Z
Update build doc for running pyspark after building with sbt
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21628: [SPARK-23776][DOC] Update instructions for running PySpa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21628
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21628: [SPARK-23776][DOC] Update instructions for runnin...
Posted by bersprockets <gi...@git.apache.org>.
Github user bersprockets commented on a diff in the pull request:
https://github.com/apache/spark/pull/21628#discussion_r197667457
--- Diff: docs/building-spark.md ---
@@ -215,19 +215,23 @@ If you are building Spark for use in a Python environment and you wish to pip in
Alternatively, you can also run make-distribution with the --pip option.
-## PySpark Tests with Maven
+## PySpark Tests with Maven or SBT
If you are building PySpark and wish to run the PySpark tests you will need to build Spark with Hive support.
./build/mvn -DskipTests clean package -Phive
./python/run-tests
+If you are building PySpark with SBT and wish to run the PySpark tests, you will need to build Spark with Hive support and also build the test components:
+
+ ./build/sbt -Phive clean package
--- End diff --
I noticed that the pyspark tests were recently changed so that -Phive is no longer strictly necessary to run pyspark tests, but I decided not to address that in this update.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21628: [SPARK-23776][DOC] Update instructions for running PySpa...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21628
Merged to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21628: [SPARK-23776][DOC] Update instructions for running PySpa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21628
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21628: [SPARK-23776][DOC] Update instructions for running PySpa...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21628
**[Test build #92311 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92311/testReport)** for PR 21628 at commit [`be281e9`](https://github.com/apache/spark/commit/be281e92e9c3920cec477d0d1376e261718fd8d6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21628: [SPARK-23776][DOC] Update instructions for runnin...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/21628
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21628: [SPARK-23776][DOC] Update instructions for running PySpa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21628
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92278/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21628: [SPARK-23776][DOC] Update instructions for running PySpa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21628
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21628: [SPARK-23776][DOC] Update instructions for runnin...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21628#discussion_r197673014
--- Diff: docs/building-spark.md ---
@@ -215,19 +215,23 @@ If you are building Spark for use in a Python environment and you wish to pip in
Alternatively, you can also run make-distribution with the --pip option.
-## PySpark Tests with Maven
+## PySpark Tests with Maven or SBT
If you are building PySpark and wish to run the PySpark tests you will need to build Spark with Hive support.
./build/mvn -DskipTests clean package -Phive
./python/run-tests
+If you are building PySpark with SBT and wish to run the PySpark tests, you will need to build Spark with Hive support and also build the test components:
+
+ ./build/sbt -Phive clean package
+ ./build/sbt sql/test:compile
--- End diff --
Hm, shouldn't we better compile other tests too?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21628: [SPARK-23776][DOC] Update instructions for running PySpa...
Posted by bersprockets <gi...@git.apache.org>.
Github user bersprockets commented on the issue:
https://github.com/apache/spark/pull/21628
@HyukjinKwon Thanks for your help!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21628: [SPARK-23776][DOC] Update instructions for runnin...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21628#discussion_r197672927
--- Diff: docs/building-spark.md ---
@@ -215,19 +215,23 @@ If you are building Spark for use in a Python environment and you wish to pip in
Alternatively, you can also run make-distribution with the --pip option.
-## PySpark Tests with Maven
+## PySpark Tests with Maven or SBT
If you are building PySpark and wish to run the PySpark tests you will need to build Spark with Hive support.
./build/mvn -DskipTests clean package -Phive
./python/run-tests
+If you are building PySpark with SBT and wish to run the PySpark tests, you will need to build Spark with Hive support and also build the test components:
+
+ ./build/sbt -Phive clean package
--- End diff --
Yea, I think we don't necessarily mention it now.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21628: [SPARK-23776][DOC] Update instructions for running PySpa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21628
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21628: [SPARK-23776][DOC] Update instructions for running PySpa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21628
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92311/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21628: [SPARK-23776][DOC] Update instructions for running PySpa...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21628
**[Test build #92311 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92311/testReport)** for PR 21628 at commit [`be281e9`](https://github.com/apache/spark/commit/be281e92e9c3920cec477d0d1376e261718fd8d6).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21628: [SPARK-23776][DOC] Update instructions for running PySpa...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21628
**[Test build #92278 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92278/testReport)** for PR 21628 at commit [`9fcd05d`](https://github.com/apache/spark/commit/9fcd05d7cb52a68bea930625605013397b4989f6).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21628: [SPARK-23776][DOC] Update instructions for running PySpa...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21628
**[Test build #92278 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92278/testReport)** for PR 21628 at commit [`9fcd05d`](https://github.com/apache/spark/commit/9fcd05d7cb52a68bea930625605013397b4989f6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org