You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/11/29 06:15:55 UTC
[GitHub] [spark] AngersZhuuuu opened a new pull request #34732: [SPARK-37291][PYSPARK] PySpark create SparkSession should pass initialSessionOptions
AngersZhuuuu opened a new pull request #34732:
URL: https://github.com/apache/spark/pull/34732
### What changes were proposed in this pull request?
In this pr, when create SparkSession, we pass initialSessionOptions to SparkSession, to keep same code path with scala code.
### Why are the changes needed?
Keep same code path with scala code.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
existed UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981331159
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981412213
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50169/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981324141
**[Test build #145694 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145694/testReport)** for PR 34732 at commit [`7be6862`](https://github.com/apache/spark/commit/7be6862024c5d49e1954cb6764b87d2d6975c0d0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981325968
**[Test build #145696 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145696/testReport)** for PR 34732 at commit [`d374536`](https://github.com/apache/spark/commit/d374536c60039523df7ef81749137d85d58b9e54).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981431725
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981578165
**[Test build #145701 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145701/testReport)** for PR 34732 at commit [`4598e8b`](https://github.com/apache/spark/commit/4598e8b52e361ef10e0ec3057041b10feb01f8ce).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981325968
**[Test build #145696 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145696/testReport)** for PR 34732 at commit [`d374536`](https://github.com/apache/spark/commit/d374536c60039523df7ef81749137d85d58b9e54).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #34732:
URL: https://github.com/apache/spark/pull/34732#discussion_r758087493
##########
File path: python/pyspark/sql/session.py
##########
@@ -305,10 +305,7 @@ def __init__(
):
jsparkSession = self._jvm.SparkSession.getDefaultSession().get()
else:
- jsparkSession = self._jvm.SparkSession(self._jsc.sc())
- if options is not None:
- for key, value in options.items():
- jsparkSession.sharedState().conf().set(key, value)
+ jsparkSession = self._jvm.SparkSession(self._jsc.sc(), self._jvm.PythonUtils.toScalaMap(options))
Review comment:
> I think you can just do:
> if options is not None:
> jsparkSession = self._jvm.SparkSession(self._jsc.sc(), options)
If so, we need to add a `if else`, why not just one line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981387662
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50164/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #34732:
URL: https://github.com/apache/spark/pull/34732#discussion_r758065251
##########
File path: python/pyspark/sql/session.py
##########
@@ -305,10 +305,7 @@ def __init__(
):
jsparkSession = self._jvm.SparkSession.getDefaultSession().get()
else:
- jsparkSession = self._jvm.SparkSession(self._jsc.sc())
- if options is not None:
- for key, value in options.items():
- jsparkSession.sharedState().conf().set(key, value)
+ jsparkSession = self._jvm.SparkSession(self._jsc.sc(), self._jvm.PythonUtils.toScalaMap(options))
Review comment:
With `private[sql] def this(sc: SparkContext) = this(sc, Map.empty)` won't need such changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981324141
**[Test build #145694 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145694/testReport)** for PR 34732 at commit [`7be6862`](https://github.com/apache/spark/commit/7be6862024c5d49e1954cb6764b87d2d6975c0d0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981582559
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34732:
URL: https://github.com/apache/spark/pull/34732#discussion_r758064015
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala
##########
@@ -97,11 +97,11 @@ class SparkSession private(
* since that would cause every new session to reinvoke Spark Session Extensions on the currently
* running extensions.
*/
- private[sql] def this(sc: SparkContext) = {
+ private[sql] def this(sc: SparkContext, initialSessionOptions: Map[String, String]) = {
Review comment:
Can we have another constructor: `private[sql] def this(sc: SparkContext) = this(sc, Map.empty)` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981350687
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50167/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981348630
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50164/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981383959
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50170/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981357549
**[Test build #145701 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145701/testReport)** for PR 34732 at commit [`4598e8b`](https://github.com/apache/spark/commit/4598e8b52e361ef10e0ec3057041b10feb01f8ce).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-982248372
**[Test build #145742 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145742/testReport)** for PR 34732 at commit [`21ab18f`](https://github.com/apache/spark/commit/21ab18f8278fdab3a6ff5b7328aea6fa10083593).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981327528
**[Test build #145697 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145697/testReport)** for PR 34732 at commit [`f6df6a8`](https://github.com/apache/spark/commit/f6df6a8d4efa1f22dd280396e11b93ddaedd3e68).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981330252
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50166/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34732:
URL: https://github.com/apache/spark/pull/34732#discussion_r758064643
##########
File path: python/pyspark/sql/session.py
##########
@@ -305,10 +305,7 @@ def __init__(
):
jsparkSession = self._jvm.SparkSession.getDefaultSession().get()
else:
- jsparkSession = self._jvm.SparkSession(self._jsc.sc())
- if options is not None:
- for key, value in options.items():
- jsparkSession.sharedState().conf().set(key, value)
+ jsparkSession = self._jvm.SparkSession(self._jsc.sc(), self._jvm.PythonUtils.toScalaMap(options))
Review comment:
Hm, is `self._jvm.PythonUtils.toScalaMap` required?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981381807
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50169/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981431725
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-982248372
**[Test build #145742 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145742/testReport)** for PR 34732 at commit [`21ab18f`](https://github.com/apache/spark/commit/21ab18f8278fdab3a6ff5b7328aea6fa10083593).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981329560
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145694/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34732:
URL: https://github.com/apache/spark/pull/34732#discussion_r758065143
##########
File path: python/pyspark/sql/session.py
##########
@@ -305,10 +305,7 @@ def __init__(
):
jsparkSession = self._jvm.SparkSession.getDefaultSession().get()
else:
- jsparkSession = self._jvm.SparkSession(self._jsc.sc())
- if options is not None:
- for key, value in options.items():
- jsparkSession.sharedState().conf().set(key, value)
+ jsparkSession = self._jvm.SparkSession(self._jsc.sc(), self._jvm.PythonUtils.toScalaMap(options))
Review comment:
I think you can just do:
```python
if options is not None:
jsparkSession = self._jvm.SparkSession(self._jsc.sc(), options)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34732:
URL: https://github.com/apache/spark/pull/34732#discussion_r758258960
##########
File path: python/pyspark/sql/session.py
##########
@@ -305,10 +305,9 @@ def __init__(
):
jsparkSession = self._jvm.SparkSession.getDefaultSession().get()
else:
- jsparkSession = self._jvm.SparkSession(self._jsc.sc())
- if options is not None:
- for key, value in options.items():
- jsparkSession.sharedState().conf().set(key, value)
+ jsparkSession = self._jvm.SparkSession(
+ self._jsc.sc(), self._jvm.PythonUtils.toScalaMap(options)
Review comment:
Looks good but last question. is this to ScalaMap required? We can make the Scala side to take Java map
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981330133
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145697/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-982295028
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50212/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-982391908
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145742/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981427887
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50170/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-982330696
Merged to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34732: [SPARK-37291][PYSPARK] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981322758
ping @HyukjinKwon @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981330133
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145697/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981329560
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145694/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981330241
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50166/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #34732:
URL: https://github.com/apache/spark/pull/34732#discussion_r758896993
##########
File path: python/pyspark/sql/session.py
##########
@@ -305,10 +305,9 @@ def __init__(
):
jsparkSession = self._jvm.SparkSession.getDefaultSession().get()
else:
- jsparkSession = self._jvm.SparkSession(self._jsc.sc())
- if options is not None:
- for key, value in options.items():
- jsparkSession.sharedState().conf().set(key, value)
+ jsparkSession = self._jvm.SparkSession(
+ self._jsc.sc(), self._jvm.PythonUtils.toScalaMap(options)
Review comment:
> Looks good but last question. is this to ScalaMap required? We can make the Scala side to take Java map
It seems worthwhile to use Java map to reduce one call of py4j. Although Java map here looks not consistent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981331159
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145696/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #34732:
URL: https://github.com/apache/spark/pull/34732#discussion_r758065251
##########
File path: python/pyspark/sql/session.py
##########
@@ -305,10 +305,7 @@ def __init__(
):
jsparkSession = self._jvm.SparkSession.getDefaultSession().get()
else:
- jsparkSession = self._jvm.SparkSession(self._jsc.sc())
- if options is not None:
- for key, value in options.items():
- jsparkSession.sharedState().conf().set(key, value)
+ jsparkSession = self._jvm.SparkSession(self._jsc.sc(), self._jvm.PythonUtils.toScalaMap(options))
Review comment:
With `private[sql] def this(sc: SparkContext) = this(sc, Map.empty)` won't need such changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981331123
**[Test build #145696 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145696/testReport)** for PR 34732 at commit [`d374536`](https://github.com/apache/spark/commit/d374536c60039523df7ef81749137d85d58b9e54).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981330110
**[Test build #145697 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145697/testReport)** for PR 34732 at commit [`f6df6a8`](https://github.com/apache/spark/commit/f6df6a8d4efa1f22dd280396e11b93ddaedd3e68).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981330252
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50166/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981351647
**[Test build #145699 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145699/testReport)** for PR 34732 at commit [`b33d254`](https://github.com/apache/spark/commit/b33d254c8b7cd892e350058645fb8a4d5365cc23).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981357549
**[Test build #145701 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145701/testReport)** for PR 34732 at commit [`4598e8b`](https://github.com/apache/spark/commit/4598e8b52e361ef10e0ec3057041b10feb01f8ce).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981351647
**[Test build #145699 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145699/testReport)** for PR 34732 at commit [`b33d254`](https://github.com/apache/spark/commit/b33d254c8b7cd892e350058645fb8a4d5365cc23).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-982391908
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145742/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-982269964
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50212/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981329520
**[Test build #145694 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145694/testReport)** for PR 34732 at commit [`7be6862`](https://github.com/apache/spark/commit/7be6862024c5d49e1954cb6764b87d2d6975c0d0).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981325158
Thanks for the followup!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-982295992
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50212/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-982295992
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50212/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981387662
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50164/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-982386226
**[Test build #145742 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145742/testReport)** for PR 34732 at commit [`21ab18f`](https://github.com/apache/spark/commit/21ab18f8278fdab3a6ff5b7328aea6fa10083593).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981332981
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50167/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #34732:
URL: https://github.com/apache/spark/pull/34732#discussion_r758086614
##########
File path: python/pyspark/sql/session.py
##########
@@ -305,10 +305,7 @@ def __init__(
):
jsparkSession = self._jvm.SparkSession.getDefaultSession().get()
else:
- jsparkSession = self._jvm.SparkSession(self._jsc.sc())
- if options is not None:
- for key, value in options.items():
- jsparkSession.sharedState().conf().set(key, value)
+ jsparkSession = self._jvm.SparkSession(self._jsc.sc(), self._jvm.PythonUtils.toScalaMap(options))
Review comment:
> Hm, is `self._jvm.PythonUtils.toScalaMap` required?
It's required
```
org.apache.spark.sql.SparkSession([class org.apache.spark.SparkContext, class java.util.HashMap]) does not exist
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981375537
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50164/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981561714
**[Test build #145699 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145699/testReport)** for PR 34732 at commit [`b33d254`](https://github.com/apache/spark/commit/b33d254c8b7cd892e350058645fb8a4d5365cc23).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34732:
URL: https://github.com/apache/spark/pull/34732#issuecomment-981582564
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #34732: [SPARK-37291][PYSPARK][FOLLOWUP] PySpark create SparkSession should pass initialSessionOptions
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #34732:
URL: https://github.com/apache/spark/pull/34732
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org