You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by ueshin <gi...@git.apache.org> on 2018/09/25 11:37:19 UTC
[GitHub] spark pull request #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf fo...
GitHub user ueshin opened a pull request:
https://github.com/apache/spark/pull/22545
[SPARK-25525][SQL][PYSPARK] Do not update conf for existing SparkContext in SparkSession.getOrCreate.
## What changes were proposed in this pull request?
In [SPARK-20946](https://issues.apache.org/jira/browse/SPARK-20946), we modified `SparkSession.getOrCreate` to not update conf for existing `SparkContext` because `SparkContext` is shared by all sessions.
We should not update it in PySpark side as well.
## How was this patch tested?
Added tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ueshin/apache-spark issues/SPARK-25525/not_update_existing_conf
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22545.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22545
----
commit 52584d943c63839089293c8ceb9e0d844fe6492d
Author: Takuya UESHIN <ue...@...>
Date: 2018-09-25T11:18:12Z
Do not update conf for existing SparkContext in SparkSession.getOrCreate.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf fo...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/22545
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22545
**[Test build #96550 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96550/testReport)** for PR 22545 at commit [`52584d9`](https://github.com/apache/spark/commit/52584d943c63839089293c8ceb9e0d844fe6492d).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22545
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22545
LGTM
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf fo...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22545#discussion_r223408456
--- Diff: python/pyspark/sql/session.py ---
@@ -156,7 +156,7 @@ def getOrCreate(self):
default.
>>> s1 = SparkSession.builder.config("k1", "v1").getOrCreate()
- >>> s1.conf.get("k1") == s1.sparkContext.getConf().get("k1") == "v1"
+ >>> s1.conf.get("k1") == "v1"
--- End diff --
In that case, we might have to put the behaviour changes by https://github.com/apache/spark/pull/18536 together to the migration guide as well.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22545
**[Test build #96553 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96553/testReport)** for PR 22545 at commit [`ac0243a`](https://github.com/apache/spark/commit/ac0243af85f9c034c77eb06c72f1df5ca7c19193).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22545
@cloud-fan, do we target this 2.4? Looks it might break an existing app, in particular, when a Python shell creates a session and another shell (like Zeppelin) or another session depends on a configuration in spark context. For instance, the fixed doctest:
```python
s1 = SparkSession.builder.config("k1", "v1").getOrCreate()
>>> s1 = SparkSession.builder.config("k1", "v1").getOrCreate()
>>> s1.conf.get("k1") == s1.sparkContext.getConf().get("k1") == "v1"
```
Looks like a bug but the behaviour change by this looks potentially quite crucial.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22545
The scala side change is already in 2.3.0. If we are ok with the behavior inconsistency between python and scala, it's fine to merge it to master only (and revert #22552 from 2.4 as well).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22545
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22545
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22545
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/22545
I'm okay with merging this only into master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22545
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3514/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22545
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96550/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/22545
@dongjoon-hyun I'm sorry that I haven't shared the context, but I talked with @cloud-fan about https://github.com/apache/spark/pull/22545#discussion_r220414608 off-line, and he wanted to separate the `HiveContext` fix into another pr, so I submitted #22552.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/22545
cc @cloud-fan
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22545
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3435/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22545
**[Test build #96663 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96663/testReport)** for PR 22545 at commit [`2ff180e`](https://github.com/apache/spark/commit/2ff180e1a2aaa9196f6d9ef2478b3834164ccbeb).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf fo...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22545#discussion_r220414608
--- Diff: python/pyspark/sql/context.py ---
@@ -485,7 +485,8 @@ def __init__(self, sparkContext, jhiveContext=None):
"SparkSession.builder.enableHiveSupport().getOrCreate() instead.",
DeprecationWarning)
if jhiveContext is None:
- sparkSession = SparkSession.builder.enableHiveSupport().getOrCreate()
+ sparkContext._conf.set("spark.sql.catalogImplementation", "hive")
+ sparkSession = SparkSession.builder._sparkContext(sparkContext).getOrCreate()
--- End diff --
why this change?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22545
Anyway, merged to master.
Let me take #22552 out of branch-2.4 for now but please feel free to get this and that into branch-2.4 (without checking along with me) if you guys feel strongly.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22545
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96663/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22545
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96553/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22545
**[Test build #96550 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96550/testReport)** for PR 22545 at commit [`52584d9`](https://github.com/apache/spark/commit/52584d943c63839089293c8ceb9e0d844fe6492d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22545
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22545
SGTM
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22545
**[Test build #96663 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96663/testReport)** for PR 22545 at commit [`2ff180e`](https://github.com/apache/spark/commit/2ff180e1a2aaa9196f6d9ef2478b3834164ccbeb).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22545
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3508/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22545
**[Test build #96655 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96655/testReport)** for PR 22545 at commit [`d8c60cb`](https://github.com/apache/spark/commit/d8c60cb1b0206740233f1ccdfd8c35b443869e1f).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22545
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96655/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22545
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf fo...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/22545#discussion_r223415446
--- Diff: python/pyspark/sql/session.py ---
@@ -156,7 +156,7 @@ def getOrCreate(self):
default.
>>> s1 = SparkSession.builder.config("k1", "v1").getOrCreate()
- >>> s1.conf.get("k1") == s1.sparkContext.getConf().get("k1") == "v1"
+ >>> s1.conf.get("k1") == "v1"
--- End diff --
We can do it together.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22545
**[Test build #96655 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96655/testReport)** for PR 22545 at commit [`d8c60cb`](https://github.com/apache/spark/commit/d8c60cb1b0206740233f1ccdfd8c35b443869e1f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22545
**[Test build #96553 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96553/testReport)** for PR 22545 at commit [`ac0243a`](https://github.com/apache/spark/commit/ac0243af85f9c034c77eb06c72f1df5ca7c19193).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22545
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22545
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf fo...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/22545#discussion_r223397858
--- Diff: python/pyspark/sql/session.py ---
@@ -156,7 +156,7 @@ def getOrCreate(self):
default.
>>> s1 = SparkSession.builder.config("k1", "v1").getOrCreate()
- >>> s1.conf.get("k1") == s1.sparkContext.getConf().get("k1") == "v1"
+ >>> s1.conf.get("k1") == "v1"
--- End diff --
@ueshin Could we also update the migration guide about this change?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22545
I think the session support is kind of partially implemented in Python side, and not being very well tested. There are some inconsistency between Python and Scala side (for instance see https://github.com/apache/spark/pull/21990). I was also thinking of targeting this one only into master.
If the release is not quite close and the code here's well tested and implemented, I'd be happy with going to branch-2.4 but .. if you guys are fine, I hope we can target this only into master.
WDYT, @ueshin?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf for exist...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22545
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3434/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf fo...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22545#discussion_r220784330
--- Diff: python/pyspark/sql/session.py ---
@@ -181,17 +181,11 @@ def getOrCreate(self):
sparkConf.set(key, value)
sc = SparkContext.getOrCreate(sparkConf)
# This SparkContext may be an existing one.
--- End diff --
tiny nit: can we move this comment above `sc = ...`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22545: [SPARK-25525][SQL][PYSPARK] Do not update conf fo...
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22545#discussion_r223953269
--- Diff: python/pyspark/sql/session.py ---
@@ -156,7 +156,7 @@ def getOrCreate(self):
default.
>>> s1 = SparkSession.builder.config("k1", "v1").getOrCreate()
- >>> s1.conf.get("k1") == s1.sparkContext.getConf().get("k1") == "v1"
+ >>> s1.conf.get("k1") == "v1"
--- End diff --
Submitted a pr to update the migration guide #22682.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org