You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by RussellSpitzer <gi...@git.apache.org> on 2018/08/03 15:57:38 UTC
[GitHub] spark pull request #21988: [SPARK-25003][PYSPARK] Use SessionExtensions in P...
GitHub user RussellSpitzer opened a pull request:
https://github.com/apache/spark/pull/21988
[SPARK-25003][PYSPARK] Use SessionExtensions in Pyspark
## What changes were proposed in this pull request?
Previously Pyspark used the private constructor for SparkSession when
building that object. This resulted in a SparkSession without checking
the sql.extensions parameter for additional session extensions. To fix
this we instead use the Session.builder() path as SparkR uses, this
loads the extensions and allows their use in PySpark.
## How was this patch tested?
This was manually tested by passing a class to spark.sql.extensions and making sure it's included strategies appeared in the spark._jsparkSession.sessionState.planner.strategies list. We could add a automatic test but i'm not very familiar with the Pyspark Testing framework. But I would be glad to implement that if requested.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/RussellSpitzer/spark SPARK-25003-branch-2.2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21988.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21988
----
commit 9c6e0671bf7311e2647e774c8d247c43037fc12c
Author: Russell Spitzer <ru...@...>
Date: 2018-08-03T15:48:15Z
[SPARK-25003][PYSPARK] Use SessionExtensions in Pyspark
Previously Pyspark used the private constructor for SparkSession when
building that object. This resulted in a SparkSession without checking
the sql.extensions parameter for additional session extensions. To fix
this we instead use the Session.builder() path as SparkR uses, this
loads the extensions and allows their use in PySpark.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21988: [SPARK-25003][PYSPARK] Use SessionExtensions in Pyspark
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21988
**[Test build #94153 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94153/testReport)** for PR 21988 at commit [`64bcb6f`](https://github.com/apache/spark/commit/64bcb6f710f12b83fbd0172d4b096a47f86dcdea).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21988: [SPARK-25003][PYSPARK] Use SessionExtensions in Pyspark
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21988
**[Test build #94153 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94153/testReport)** for PR 21988 at commit [`64bcb6f`](https://github.com/apache/spark/commit/64bcb6f710f12b83fbd0172d4b096a47f86dcdea).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21988: [SPARK-25003][PYSPARK][BRANCH-2.2] Use SessionExtensions...
Posted by RussellSpitzer <gi...@git.apache.org>.
Github user RussellSpitzer commented on the issue:
https://github.com/apache/spark/pull/21988
@felixcheung I just didn't know what version to target so I made a a PR for each one. We can just close the ones that shouldn't be merged.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21988: [SPARK-25003][PYSPARK] Use SessionExtensions in Pyspark
Posted by RussellSpitzer <gi...@git.apache.org>.
Github user RussellSpitzer commented on the issue:
https://github.com/apache/spark/pull/21988
Local PEP didn't seem to mind this code ... Fixed up the indentation so hopefully jenkins will like it now
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21988: [SPARK-25003][PYSPARK] Use SessionExtensions in Pyspark
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21988
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21988: [SPARK-25003][PYSPARK] Use SessionExtensions in Pyspark
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21988
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21988: [SPARK-25003][PYSPARK] Use SessionExtensions in Pyspark
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21988
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21988: [SPARK-25003][PYSPARK][BRANCH-2.2] Use SessionExt...
Posted by RussellSpitzer <gi...@git.apache.org>.
Github user RussellSpitzer closed the pull request at:
https://github.com/apache/spark/pull/21988
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21988: [SPARK-25003][PYSPARK] Use SessionExtensions in Pyspark
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21988
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94153/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21988: [SPARK-25003][PYSPARK] Use SessionExtensions in Pyspark
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/21988
why do we need multiple PRs? typically we do that only then the change is non-trivial or cannot be backported by cherry-picking.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21988: [SPARK-25003][PYSPARK] Use SessionExtensions in Pyspark
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21988
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94144/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21988: [SPARK-25003][PYSPARK] Use SessionExtensions in Pyspark
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21988
And put `[BRANCH-2.3]` into the title of the PR #21989 too?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21988: [SPARK-25003][PYSPARK] Use SessionExtensions in Pyspark
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21988
**[Test build #94144 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94144/testReport)** for PR 21988 at commit [`9c6e067`](https://github.com/apache/spark/commit/9c6e0671bf7311e2647e774c8d247c43037fc12c).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21988: [SPARK-25003][PYSPARK][BRANCH-2.2] Use SessionExtensions...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/21988
we always open against master and backport if agreed upon.
this is documented here https://spark.apache.org/contributing.html
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21988: [SPARK-25003][PYSPARK] Use SessionExtensions in Pyspark
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21988
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21988: [SPARK-25003][PYSPARK] Use SessionExtensions in Pyspark
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21988
@RussellSpitzer, btw mind if I ask to put `[BRANCH-2.2]` inthe PR title to just make it less confused?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21988: [SPARK-25003][PYSPARK] Use SessionExtensions in Pyspark
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21988
**[Test build #94144 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94144/testReport)** for PR 21988 at commit [`9c6e067`](https://github.com/apache/spark/commit/9c6e0671bf7311e2647e774c8d247c43037fc12c).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21988: [SPARK-25003][PYSPARK][BRANCH-2.2] Use SessionExtensions...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21988
Yea, let's just close except the master one.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org