You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/11/11 15:09:50 UTC
[GitHub] [spark] AngersZhuuuu opened a new pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
AngersZhuuuu opened a new pull request #34559:
URL: https://github.com/apache/spark/pull/34559
### What changes were proposed in this pull request?
When use write pyspark script like
```
conf = SparkConf().setAppName("test")
sc = SparkContext(conf = conf)
session = SparkSession().build().enableHiveSupport().getOrCreate()
```
It will build a session without hive support since we use a existed SparkContext and we create SparkSession use
```
SparkSession(sc)
```
This cause we loss configuration added by `config()` such as catalog implement.
In scala SparkSession, we create SparkSession with SparkContext and option configuration and will path option configuration to SharedState then use SharedState create SessionState, but in pyspark, we won't pass options configuration to shared state, but pass to SessionState, but this time SessionState has been initialized. So it won't support hive.
In this pr, I pass option configurations to SharedState, then when init SessionState, this options will be passed to SharedState too.
### Why are the changes needed?
Avoid loss configuration when build SparkSession in pyspark
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Manuel tested
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966823651
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49605/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966851498
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #34559: [SPARK-37291][PYTHON][SQL] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #34559:
URL: https://github.com/apache/spark/pull/34559#discussion_r758003899
##########
File path: python/pyspark/sql/session.py
##########
@@ -301,6 +306,9 @@ def __init__(self, sparkContext: SparkContext, jsparkSession: Optional[JavaObjec
jsparkSession = self._jvm.SparkSession.getDefaultSession().get()
else:
jsparkSession = self._jvm.SparkSession(self._jsc.sc())
Review comment:
> Can we actually leverage existing constructor on SparkSession to pass the initial options instead of setting it manually? Here unlike Scala, it initiates sharedState always. I think it's best to keep the code path matched.
Yea, will try this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966851498
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966890385
**[Test build #145148 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145148/testReport)** for PR 34559 at commit [`aafb6a1`](https://github.com/apache/spark/commit/aafb6a1f31789a325bfb50c9b1fef55607f6df62).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966825955
**[Test build #145143 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145143/testReport)** for PR 34559 at commit [`7f1e668`](https://github.com/apache/spark/commit/7f1e66843ae151d7087c4ce9eb684c34b67e0f54).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966875657
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-967566774
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49643/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34559: [SPARK-37291][PYTHON][SQL] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34559:
URL: https://github.com/apache/spark/pull/34559#discussion_r748779639
##########
File path: python/pyspark/sql/session.py
##########
@@ -287,7 +287,12 @@ def getOrCreate(self) -> "SparkSession":
_instantiatedSession: ClassVar[Optional["SparkSession"]] = None
_activeSession: ClassVar[Optional["SparkSession"]] = None
- def __init__(self, sparkContext: SparkContext, jsparkSession: Optional[JavaObject] = None):
+ def __init__(
+ self,
+ sparkContext: SparkContext,
+ jsparkSession: Optional[JavaObject] = None,
+ options: Optional[Dict[str, Any]] = None,
+ ):
Review comment:
Yeah I don't think this is breaking. Let me double check closely by tmr EODbut from a coursory look it seems fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34559: [SPARK-37291][PYTHON][SQL] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34559:
URL: https://github.com/apache/spark/pull/34559#discussion_r748779639
##########
File path: python/pyspark/sql/session.py
##########
@@ -287,7 +287,12 @@ def getOrCreate(self) -> "SparkSession":
_instantiatedSession: ClassVar[Optional["SparkSession"]] = None
_activeSession: ClassVar[Optional["SparkSession"]] = None
- def __init__(self, sparkContext: SparkContext, jsparkSession: Optional[JavaObject] = None):
+ def __init__(
+ self,
+ sparkContext: SparkContext,
+ jsparkSession: Optional[JavaObject] = None,
+ options: Optional[Dict[str, Any]] = None,
+ ):
Review comment:
Yeah I don't think this is breaking. Let me double check closely by tmr EOD but from a coursory look it seems fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34559:
URL: https://github.com/apache/spark/pull/34559#discussion_r748777223
##########
File path: python/pyspark/sql/session.py
##########
@@ -287,7 +287,12 @@ def getOrCreate(self) -> "SparkSession":
_instantiatedSession: ClassVar[Optional["SparkSession"]] = None
_activeSession: ClassVar[Optional["SparkSession"]] = None
- def __init__(self, sparkContext: SparkContext, jsparkSession: Optional[JavaObject] = None):
+ def __init__(
+ self,
+ sparkContext: SparkContext,
+ jsparkSession: Optional[JavaObject] = None,
+ options: Optional[Dict[str, Any]] = None,
+ ):
Review comment:
This seems to be not a breaking change in Python, right? How do you think about this, @HyukjinKwon ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966785487
**[Test build #145131 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145131/testReport)** for PR 34559 at commit [`b5d995f`](https://github.com/apache/spark/commit/b5d995f1be6526c06983741e484241b08bb818cf).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966898566
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49619/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966475719
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49582/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966941094
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49619/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966810900
**[Test build #145141 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145141/testReport)** for PR 34559 at commit [`a71c903`](https://github.com/apache/spark/commit/a71c90354e0a69d3f9d60e3236324481796525b8).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966809120
**[Test build #145140 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145140/testReport)** for PR 34559 at commit [`ee3e2b3`](https://github.com/apache/spark/commit/ee3e2b3bdb1c802f5e86d9c04fea964d45ebc797).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966867110
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49614/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-967353271
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145172/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-967331263
gentle ping @dongjoon-hyun @HyukjinKwon Unit test added to confirm this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-967465585
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145174/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966415595
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49582/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966475719
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49582/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966785487
**[Test build #145131 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145131/testReport)** for PR 34559 at commit [`b5d995f`](https://github.com/apache/spark/commit/b5d995f1be6526c06983741e484241b08bb818cf).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966798147
**[Test build #145136 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145136/testReport)** for PR 34559 at commit [`e599f57`](https://github.com/apache/spark/commit/e599f57cb0c341b8a60b1512ef76acc804f4f703).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966798992
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49600/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966809517
**[Test build #145140 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145140/testReport)** for PR 34559 at commit [`ee3e2b3`](https://github.com/apache/spark/commit/ee3e2b3bdb1c802f5e86d9c04fea964d45ebc797).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966809531
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145140/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34559: [SPARK-37291][PYTHON][SQL] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34559:
URL: https://github.com/apache/spark/pull/34559#discussion_r758001741
##########
File path: python/pyspark/sql/session.py
##########
@@ -301,6 +306,9 @@ def __init__(self, sparkContext: SparkContext, jsparkSession: Optional[JavaObjec
jsparkSession = self._jvm.SparkSession.getDefaultSession().get()
else:
jsparkSession = self._jvm.SparkSession(self._jsc.sc())
Review comment:
LGTM with a couple of nits: @AngersZhuuuu,
- Can we actually leverage existing constructor on `SparkSession` to pass the initial options instead of setting it manually? Here unlike Scala, it initiates `sharedState` always. I think it's best to keep the code path matched.
- Another nit is that: It's always preferred to use less Py4J connections which exposes potential flakiness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34559: [SPARK-37291][PYTHON][SQL] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34559:
URL: https://github.com/apache/spark/pull/34559#discussion_r758008380
##########
File path: python/pyspark/sql/session.py
##########
@@ -301,6 +306,9 @@ def __init__(self, sparkContext: SparkContext, jsparkSession: Optional[JavaObjec
jsparkSession = self._jvm.SparkSession.getDefaultSession().get()
else:
jsparkSession = self._jvm.SparkSession(self._jsc.sc())
Review comment:
Yup, let's create a followup PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu edited a comment on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu edited a comment on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966772039
> PySpark GitHub Action jobs seem to complain. Could you take a look at that, @AngersZhuuuu ?
mis write sharedState ==. Also update the pr desc make it more clear
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966791714
**[Test build #145134 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145134/testReport)** for PR 34559 at commit [`05c65a4`](https://github.com/apache/spark/commit/05c65a4e40790c3aeb18010c58da14b6d685a130).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966791355
**[Test build #145134 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145134/testReport)** for PR 34559 at commit [`05c65a4`](https://github.com/apache/spark/commit/05c65a4e40790c3aeb18010c58da14b6d685a130).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966804905
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966905438
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145148/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-967353271
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145172/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966849011
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49614/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-967445915
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49643/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966850630
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49611/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966809120
**[Test build #145140 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145140/testReport)** for PR 34559 at commit [`ee3e2b3`](https://github.com/apache/spark/commit/ee3e2b3bdb1c802f5e86d9c04fea964d45ebc797).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966808134
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49605/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966905438
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145148/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966768240
**[Test build #145129 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145129/testReport)** for PR 34559 at commit [`1518b2d`](https://github.com/apache/spark/commit/1518b2dc81b2fdab276db93d60407f1d2367bc0f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966785926
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145131/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-967566774
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49643/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966868828
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49611/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966875654
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34559: [SPARK-37291][PYTHON][SQL] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34559:
URL: https://github.com/apache/spark/pull/34559#discussion_r758005218
##########
File path: python/pyspark/sql/session.py
##########
@@ -301,6 +306,9 @@ def __init__(self, sparkContext: SparkContext, jsparkSession: Optional[JavaObjec
jsparkSession = self._jvm.SparkSession.getDefaultSession().get()
else:
jsparkSession = self._jvm.SparkSession(self._jsc.sc())
Review comment:
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966382467
**[Test build #145114 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145114/testReport)** for PR 34559 at commit [`54e3a0c`](https://github.com/apache/spark/commit/54e3a0c5afd5554d69687078a2b4969a0437c85e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-967330670
**[Test build #145172 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145172/testReport)** for PR 34559 at commit [`38b96bf`](https://github.com/apache/spark/commit/38b96bf6d8b3f74207b7f5d1d69cdbda1720720a).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-967465585
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145174/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966773227
**[Test build #145129 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145129/testReport)** for PR 34559 at commit [`1518b2d`](https://github.com/apache/spark/commit/1518b2dc81b2fdab276db93d60407f1d2367bc0f).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966810357
**[Test build #145141 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145141/testReport)** for PR 34559 at commit [`a71c903`](https://github.com/apache/spark/commit/a71c90354e0a69d3f9d60e3236324481796525b8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966878738
**[Test build #145148 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145148/testReport)** for PR 34559 at commit [`aafb6a1`](https://github.com/apache/spark/commit/aafb6a1f31789a325bfb50c9b1fef55607f6df62).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #34559: [SPARK-37291][PYTHON][SQL] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #34559:
URL: https://github.com/apache/spark/pull/34559
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966792501
**[Test build #145136 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145136/testReport)** for PR 34559 at commit [`e599f57`](https://github.com/apache/spark/commit/e599f57cb0c341b8a60b1512ef76acc804f4f703).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966768240
**[Test build #145129 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145129/testReport)** for PR 34559 at commit [`1518b2d`](https://github.com/apache/spark/commit/1518b2dc81b2fdab276db93d60407f1d2367bc0f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966785926
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145131/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966825955
**[Test build #145143 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145143/testReport)** for PR 34559 at commit [`7f1e668`](https://github.com/apache/spark/commit/7f1e66843ae151d7087c4ce9eb684c34b67e0f54).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966848235
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #34559: [SPARK-37291][PYTHON][SQL] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #34559:
URL: https://github.com/apache/spark/pull/34559#discussion_r758006371
##########
File path: python/pyspark/sql/session.py
##########
@@ -301,6 +306,9 @@ def __init__(self, sparkContext: SparkContext, jsparkSession: Optional[JavaObjec
jsparkSession = self._jvm.SparkSession.getDefaultSession().get()
else:
jsparkSession = self._jvm.SparkSession(self._jsc.sc())
Review comment:
>
Create a new one or a followup?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-967358775
**[Test build #145174 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145174/testReport)** for PR 34559 at commit [`1185d1f`](https://github.com/apache/spark/commit/1185d1fe1836a90ca2b94aed8499eeb4ef232609).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-967412338
**[Test build #145174 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145174/testReport)** for PR 34559 at commit [`1185d1f`](https://github.com/apache/spark/commit/1185d1fe1836a90ca2b94aed8499eeb4ef232609).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-967546343
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49643/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966393659
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145114/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966784679
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145129/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966784679
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145129/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966791724
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966824758
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49602/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966941094
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49619/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966791355
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966865692
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49612/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-967329930
**[Test build #145172 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145172/testReport)** for PR 34559 at commit [`38b96bf`](https://github.com/apache/spark/commit/38b96bf6d8b3f74207b7f5d1d69cdbda1720720a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966825185
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-967358775
**[Test build #145174 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145174/testReport)** for PR 34559 at commit [`1185d1f`](https://github.com/apache/spark/commit/1185d1fe1836a90ca2b94aed8499eeb4ef232609).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966825183
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966934136
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49619/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966393370
**[Test build #145114 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145114/testReport)** for PR 34559 at commit [`54e3a0c`](https://github.com/apache/spark/commit/54e3a0c5afd5554d69687078a2b4969a0437c85e).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966382467
**[Test build #145114 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145114/testReport)** for PR 34559 at commit [`54e3a0c`](https://github.com/apache/spark/commit/54e3a0c5afd5554d69687078a2b4969a0437c85e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966393659
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145114/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34559: [SPARK-37291][PYTHON][SQL] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34559:
URL: https://github.com/apache/spark/pull/34559#discussion_r748779639
##########
File path: python/pyspark/sql/session.py
##########
@@ -287,7 +287,12 @@ def getOrCreate(self) -> "SparkSession":
_instantiatedSession: ClassVar[Optional["SparkSession"]] = None
_activeSession: ClassVar[Optional["SparkSession"]] = None
- def __init__(self, sparkContext: SparkContext, jsparkSession: Optional[JavaObject] = None):
+ def __init__(
+ self,
+ sparkContext: SparkContext,
+ jsparkSession: Optional[JavaObject] = None,
+ options: Optional[Dict[str, Any]] = None,
+ ):
Review comment:
Yeah I don't think this is breaking. Let me double check closely by tmr EOD but from a cursory look it seems fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966475681
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49582/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966772039
> PySpark GitHub Action jobs seem to complain. Could you take a look at that, @AngersZhuuuu ?
> PySpark GitHub Action jobs seem to complain. Could you take a look at that, @AngersZhuuuu ?
mis write sharedState ==. Also update the pr desc make it more clear
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966379732
ping @dongjoon-hyun @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-967329930
**[Test build #145172 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145172/testReport)** for PR 34559 at commit [`38b96bf`](https://github.com/apache/spark/commit/38b96bf6d8b3f74207b7f5d1d69cdbda1720720a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966791724
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145134/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966837260
**[Test build #145143 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145143/testReport)** for PR 34559 at commit [`7f1e668`](https://github.com/apache/spark/commit/7f1e66843ae151d7087c4ce9eb684c34b67e0f54).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966849062
@dongjoon-hyun tested a lot since not familiar with python code. Now I think can be reviewed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966810357
**[Test build #145141 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145141/testReport)** for PR 34559 at commit [`a71c903`](https://github.com/apache/spark/commit/a71c90354e0a69d3f9d60e3236324481796525b8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966810921
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145141/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966809531
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145140/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966810921
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145141/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966803849
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49602/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966849936
Thank you for updates, @AngersZhuuuu .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966785915
**[Test build #145131 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145131/testReport)** for PR 34559 at commit [`b5d995f`](https://github.com/apache/spark/commit/b5d995f1be6526c06983741e484241b08bb818cf).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966784004
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49600/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966823061
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49607/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34559: [SPARK-37291][SQL][PYSPARK] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34559:
URL: https://github.com/apache/spark/pull/34559#issuecomment-966878738
**[Test build #145148 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145148/testReport)** for PR 34559 at commit [`aafb6a1`](https://github.com/apache/spark/commit/aafb6a1f31789a325bfb50c9b1fef55607f6df62).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34559: [SPARK-37291][PYTHON][SQL] PySpark init SparkSession should copy conf to sharedState
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34559:
URL: https://github.com/apache/spark/pull/34559#discussion_r758001741
##########
File path: python/pyspark/sql/session.py
##########
@@ -301,6 +306,9 @@ def __init__(self, sparkContext: SparkContext, jsparkSession: Optional[JavaObjec
jsparkSession = self._jvm.SparkSession.getDefaultSession().get()
else:
jsparkSession = self._jvm.SparkSession(self._jsc.sc())
Review comment:
LGTM with a couple of nits: @AngersZhuuuu,
- Can we actually leverage existing constructor on `SparkSession` to pass the initial options instead of manually? Here unlike Scala, it initiates `sharedState` always. I think it's best to keep the code path matched.
- Another nit is that: It's always preferred to use less Py4J connections which exposes potential flakiness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org