You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/02/17 16:58:15 UTC
[GitHub] [spark] afavaro opened a new pull request #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
afavaro opened a new pull request #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
As discussed on the Jira ticket, this change clears the SQLContext._instantiatedContext class attribute when the SparkSession is stopped. That way, the attribute will be reset with a new, usable SQLContext when a new SparkSession is started.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
When the underlying SQLContext is instantiated for a SparkSession, the instance is saved as a class attribute and returned from subsequent calls to SQLContext.getOrCreate(). If the SparkContext is stopped and a new one started, the SQLContext class attribute is never cleared so any code which calls SQLContext.getOrCreate() will get a SQLContext with a reference to the old, unusable SparkContext.
A similar issue was identified and fixed for SparkSession in [SPARK-19055](https://issues.apache.org/jira/browse/SPARK-19055), but the fix did not change SQLContext as well. I ran into this because mllib still [uses](https://github.com/apache/spark/blob/master/python/pyspark/mllib/common.py#L105) SQLContext.getOrCreate() under the hood.
### Does this PR introduce any user-facing change?
<!--
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If no, write 'No'.
-->
No
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
A new test was added. I verified that the test fails without the included change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-587120745
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118597/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-587120745
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118597/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-587086259
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#discussion_r380411871
##########
File path: python/pyspark/sql/tests/test_session.py
##########
@@ -225,6 +225,24 @@ def test_get_active_session_after_create_dataframe(self):
session2.stop()
+class SparkSessionTests4(ReusedSQLTestCase):
Review comment:
SparkSessionTests5?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
viirya commented on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-587969445
> > Do we also need to check if SparkContext is stopped when calling SQLContext.getOrCreate?
>
> It asserts that it's stopped in the Scala `SparkSession` constructor, https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala#L103. So don't know if it's necessary to check again.
Do you mean when calling `sparkSession = SparkSession(sc, jsqlContext.sparkSession())`, it asserts the given `sc` is not stopped?
I meant in `getOrCreate`, we might need to check if the associated `SparkContext` with old `SQLContext` is stopped.
Currently we only check `if cls._instantiatedContext is None:`. When `SparkContext` is stopped, and we call `SQLContext.getOrCreate` with a new `SparkContext`, because `cls._instantiatedContext` is not None, we still return it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-587086776
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-588581979
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-587108596
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23351/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-588589533
Merged to master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#discussion_r380422547
##########
File path: python/pyspark/sql/tests/test_session.py
##########
@@ -225,6 +225,24 @@ def test_get_active_session_after_create_dataframe(self):
session2.stop()
+class SparkSessionTests4(ReusedSQLTestCase):
Review comment:
@afavaro can you address this comment? I think it's good to go.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-587230056
The fix looks fine but I found another issue. `SQLContext.getOrCreate` was removed but being still referred in PySpark's side.. I will make a fix
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] BryanCutler commented on a change in pull request #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
BryanCutler commented on a change in pull request #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#discussion_r380879579
##########
File path: python/pyspark/sql/tests/test_session.py
##########
@@ -225,6 +225,24 @@ def test_get_active_session_after_create_dataframe(self):
session2.stop()
+class SparkSessionTests5(ReusedSQLTestCase):
+
+ def test_sqlcontext_with_stopped_sparkcontext(self):
+ # SPARK-30856: test that SQLContext.getOrCreate() returns a usable instance after
+ # the SparkContext is restarted.
Review comment:
It would be better to get the current instance of the `SQLContext` here and then later check the new context is not equal to the old.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] afavaro commented on a change in pull request #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
afavaro commented on a change in pull request #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#discussion_r381168087
##########
File path: python/pyspark/sql/tests/test_session.py
##########
@@ -225,6 +225,24 @@ def test_get_active_session_after_create_dataframe(self):
session2.stop()
+class SparkSessionTests5(ReusedSQLTestCase):
+
+ def test_sqlcontext_with_stopped_sparkcontext(self):
+ # SPARK-30856: test that SQLContext.getOrCreate() returns a usable instance after
+ # the SparkContext is restarted.
+ self.spark.stop()
+ sc = SparkContext('local[4]', self.sc.appName)
+ spark = SparkSession(sc) # Instantiate the underlying SQLContext
+ ctx_spark = SQLContext.getOrCreate(sc).sparkSession
+ self.assertIs(ctx_spark, spark)
+ try:
+ df = ctx_spark.createDataFrame([(1, 2)], ['c', 'c'])
+ df.collect()
+ finally:
+ spark.stop()
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-587108230
**[Test build #118597 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118597/testReport)** for PR 27610 at commit [`1a8a0a0`](https://github.com/apache/spark/commit/1a8a0a0a681ebe3b7ed56e01c30b0d1cd6345ca7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-587108590
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-588574112
**[Test build #118692 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118692/testReport)** for PR 27610 at commit [`fe55515`](https://github.com/apache/spark/commit/fe55515b12c296cd1c3672a8e9b22f94f888dd79).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-587228319
We should deprecate `SQLContext.getOrCreate` ... it's removed in Scala side.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-588574505
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-588573735
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
srowen commented on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-587107789
It looks plausible, but @viirya what do you think too?
Indeed SQLContext is more or less deprecated, but should still work while it exists.
It seems a little funny to manage its lifecycle in SparkSession, but, there is no way to stop a SQLContext from Pyspark? or at least there is no method that unsets the instantiated context, at all.
This seems like an OK place to do it, given that SparkSession does create the SQLContext too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#discussion_r381008674
##########
File path: python/pyspark/sql/tests/test_session.py
##########
@@ -225,6 +225,24 @@ def test_get_active_session_after_create_dataframe(self):
session2.stop()
+class SparkSessionTests5(ReusedSQLTestCase):
+
+ def test_sqlcontext_with_stopped_sparkcontext(self):
+ # SPARK-30856: test that SQLContext.getOrCreate() returns a usable instance after
+ # the SparkContext is restarted.
+ self.spark.stop()
+ sc = SparkContext('local[4]', self.sc.appName)
+ spark = SparkSession(sc) # Instantiate the underlying SQLContext
+ ctx_spark = SQLContext.getOrCreate(sc).sparkSession
+ self.assertIs(ctx_spark, spark)
+ try:
+ df = ctx_spark.createDataFrame([(1, 2)], ['c', 'c'])
+ df.collect()
+ finally:
+ spark.stop()
Review comment:
nit: We can assert if SQLContext._instantiatedContext is None now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] afavaro commented on a change in pull request #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
afavaro commented on a change in pull request #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#discussion_r381168699
##########
File path: python/pyspark/sql/tests/test_session.py
##########
@@ -225,6 +225,24 @@ def test_get_active_session_after_create_dataframe(self):
session2.stop()
+class SparkSessionTests5(ReusedSQLTestCase):
+
+ def test_sqlcontext_with_stopped_sparkcontext(self):
+ # SPARK-30856: test that SQLContext.getOrCreate() returns a usable instance after
+ # the SparkContext is restarted.
+ self.spark.stop()
+ sc = SparkContext('local[4]', self.sc.appName)
+ spark = SparkSession(sc) # Instantiate the underlying SQLContext
+ ctx_spark = SQLContext.getOrCreate(sc).sparkSession
Review comment:
How about this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
srowen commented on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-587107381
Jenkins test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-588574510
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23443/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] BryanCutler commented on a change in pull request #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
BryanCutler commented on a change in pull request #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#discussion_r380884084
##########
File path: python/pyspark/sql/tests/test_session.py
##########
@@ -225,6 +225,24 @@ def test_get_active_session_after_create_dataframe(self):
session2.stop()
+class SparkSessionTests5(ReusedSQLTestCase):
+
+ def test_sqlcontext_with_stopped_sparkcontext(self):
+ # SPARK-30856: test that SQLContext.getOrCreate() returns a usable instance after
+ # the SparkContext is restarted.
+ self.spark.stop()
+ sc = SparkContext('local[4]', self.sc.appName)
+ spark = SparkSession(sc) # Instantiate the underlying SQLContext
+ ctx_spark = SQLContext.getOrCreate(sc).sparkSession
Review comment:
nit: `ctx_spark` name is a little confusing, makes it seem like its the `SQLContext`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-588581989
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118692/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] afavaro commented on a change in pull request #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
afavaro commented on a change in pull request #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#discussion_r381168633
##########
File path: python/pyspark/sql/tests/test_session.py
##########
@@ -225,6 +225,24 @@ def test_get_active_session_after_create_dataframe(self):
session2.stop()
+class SparkSessionTests5(ReusedSQLTestCase):
+
+ def test_sqlcontext_with_stopped_sparkcontext(self):
+ # SPARK-30856: test that SQLContext.getOrCreate() returns a usable instance after
+ # the SparkContext is restarted.
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-588574112
**[Test build #118692 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118692/testReport)** for PR 27610 at commit [`fe55515`](https://github.com/apache/spark/commit/fe55515b12c296cd1c3672a8e9b22f94f888dd79).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-588581979
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-587230056
The fix looks fine but I found another issue. `SQLContext.getOrCreate` was removed but being still referred in PySpark's side.. I will make a fix. here https://github.com/apache/spark/pull/27614
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] afavaro commented on a change in pull request #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
afavaro commented on a change in pull request #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#discussion_r380544878
##########
File path: python/pyspark/sql/tests/test_session.py
##########
@@ -225,6 +225,24 @@ def test_get_active_session_after_create_dataframe(self):
session2.stop()
+class SparkSessionTests4(ReusedSQLTestCase):
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-587086259
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-588574505
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-587120736
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-587120389
**[Test build #118597 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118597/testReport)** for PR 27610 at commit [`1a8a0a0`](https://github.com/apache/spark/commit/1a8a0a0a681ebe3b7ed56e01c30b0d1cd6345ca7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] afavaro commented on a change in pull request #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
afavaro commented on a change in pull request #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#discussion_r381167953
##########
File path: python/pyspark/sql/session.py
##########
@@ -699,12 +699,14 @@ def streams(self):
def stop(self):
"""Stop the underlying :class:`SparkContext`.
"""
+ from pyspark.sql.context import SQLContext
self._sc.stop()
# We should clean the default session up. See SPARK-23228.
self._jvm.SparkSession.clearDefaultSession()
self._jvm.SparkSession.clearActiveSession()
SparkSession._instantiatedSession = None
SparkSession._activeSession = None
+ SQLContext._instantiatedContext = None
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] BryanCutler commented on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
BryanCutler commented on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-587682282
> Do we also need to check if SparkContext is stopped when calling SQLContext.getOrCreate?
It asserts that it's stopped in the Scala `SparkSession` constructor, https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala#L103. So don't know if it's necessary to check again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-587120736
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-587086776
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-588574510
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23443/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-587108230
**[Test build #118597 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118597/testReport)** for PR 27610 at commit [`1a8a0a0`](https://github.com/apache/spark/commit/1a8a0a0a681ebe3b7ed56e01c30b0d1cd6345ca7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-588581989
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118692/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-587108596
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23351/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#discussion_r381072105
##########
File path: python/pyspark/sql/session.py
##########
@@ -699,12 +699,14 @@ def streams(self):
def stop(self):
"""Stop the underlying :class:`SparkContext`.
"""
+ from pyspark.sql.context import SQLContext
self._sc.stop()
# We should clean the default session up. See SPARK-23228.
self._jvm.SparkSession.clearDefaultSession()
self._jvm.SparkSession.clearActiveSession()
SparkSession._instantiatedSession = None
SparkSession._activeSession = None
+ SQLContext._instantiatedContext = None
Review comment:
@afavaro can you remove [this line](https://github.com/apache/spark/pull/27614/files#diff-fadcca87874ab0f0c00d3c4a08d56a77R273) too after syncing to the master?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-588581669
**[Test build #118692 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118692/testReport)** for PR 27610 at commit [`fe55515`](https://github.com/apache/spark/commit/fe55515b12c296cd1c3672a8e9b22f94f888dd79).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27610:
[SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext
is restarted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27610: [SPARK-30856][SQL][PySpark] Fix SQLContext.getOrCreate() when SparkContext is restarted
URL: https://github.com/apache/spark/pull/27610#issuecomment-587108590
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org