You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/11/04 11:29:18 UTC
[GitHub] [spark] li36909 opened a new pull request #30248: [SPARK-33339][PYTHON] Pyspark application maybe hangup because of wor…
li36909 opened a new pull request #30248:
URL: https://github.com/apache/spark/pull/30248
…ker exit
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] li36909 commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
li36909 commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-725260785
> It has a conflict in branch-2.4. It's sort of a corner case so I think we don't bother porting it back. @li36909, please go ahead and open a PR to backport if you're willing to do.
ok, I will open a PR at branch-2.4, thank you!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-722280018
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-722296694
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35256/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-724559424
**[Test build #130854 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130854/testReport)** for PR 30248 at commit [`e5c33f0`](https://github.com/apache/spark/commit/e5c33f0b28e58110de04e9dbc6e1d75bc59d53f0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-722303277
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-724551786
**[Test build #130852 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130852/testReport)** for PR 30248 at commit [`8863dbb`](https://github.com/apache/spark/commit/8863dbbeaf5a20e283e7ebd92c2c55bc3a950998).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #30248:
URL: https://github.com/apache/spark/pull/30248#discussion_r520266619
##########
File path: python/pyspark/worker.py
##########
@@ -604,7 +604,7 @@ def process():
# reuse.
TaskContext._setTaskContext(None)
BarrierTaskContext._setTaskContext(None)
- except Exception:
+ except BaseException:
Review comment:
Can you change `except Exception:` at 621 line to `except BaseException` too?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-724559424
**[Test build #130854 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130854/testReport)** for PR 30248 at commit [`e5c33f0`](https://github.com/apache/spark/commit/e5c33f0b28e58110de04e9dbc6e1d75bc59d53f0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] li36909 commented on a change in pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
li36909 commented on a change in pull request #30248:
URL: https://github.com/apache/spark/pull/30248#discussion_r520379895
##########
File path: python/pyspark/worker.py
##########
@@ -604,7 +604,7 @@ def process():
# reuse.
TaskContext._setTaskContext(None)
BarrierTaskContext._setTaskContext(None)
- except Exception:
+ except BaseException:
Review comment:
updated
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-724551786
**[Test build #130852 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130852/testReport)** for PR 30248 at commit [`8863dbb`](https://github.com/apache/spark/commit/8863dbbeaf5a20e283e7ebd92c2c55bc3a950998).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] li36909 commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception
Posted by GitBox <gi...@apache.org>.
li36909 commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-722330855
@HyukjinKwon @wzhfy
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] li36909 commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application maybe hangup because of wor…
Posted by GitBox <gi...@apache.org>.
li36909 commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-722200121
> @li36909 Could you update the description? Currently it's misleading.
>
> @HyukjinKwon Actually `system.exit` can happen when user writes some incorrect code (e.g. try to create a SparkContext in a executor task while no SPARK_HOME is available on that node, or the udf case @li36909 mentioned).
> We have found several such cases from unskilled PySpark customers.
>
> It would be better for Spark to throw an exception to the user, rather than hanging and making users not know what to do.
updated
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wzhfy commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application maybe hangup because of wor…
Posted by GitBox <gi...@apache.org>.
wzhfy commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-722185651
@li36909 Could you update the description? Currently it's misleading.
@HyukjinKwon Actually `system.exit` is throw because user writes some incorrect code (e.g. try to create a SparkContext in a executor task while no SPARK_HOME is available on that node, or the udf case @li36909 mentioned). We have found several such cases from unskilled Spark customers.
It would be better for Spark to throw an exception to the user, rather than hanging and making users not know what to do.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-722263842
**[Test build #130646 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130646/testReport)** for PR 30248 at commit [`fb8d325`](https://github.com/apache/spark/commit/fb8d325c7241373ae4d2d66bb0ed2e2a87ff3a17).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-724616683
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-724576917
**[Test build #130854 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130854/testReport)** for PR 30248 at commit [`e5c33f0`](https://github.com/apache/spark/commit/e5c33f0b28e58110de04e9dbc6e1d75bc59d53f0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-724552570
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130852/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-724577514
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-724618468
Merged to master, branch-3.0 and branch-2.4.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-724616683
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-722263842
**[Test build #130646 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130646/testReport)** for PR 30248 at commit [`fb8d325`](https://github.com/apache/spark/commit/fb8d325c7241373ae4d2d66bb0ed2e2a87ff3a17).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] li36909 commented on a change in pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
li36909 commented on a change in pull request #30248:
URL: https://github.com/apache/spark/pull/30248#discussion_r520359601
##########
File path: python/pyspark/worker.py
##########
@@ -604,7 +604,7 @@ def process():
# reuse.
TaskContext._setTaskContext(None)
BarrierTaskContext._setTaskContext(None)
- except Exception:
+ except BaseException:
Review comment:
at 621 line seems no need to cache BaseException, because daemon.py will cache SystemExit at 75 line
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] li36909 commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application maybe hangup because of wor…
Posted by GitBox <gi...@apache.org>.
li36909 commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-722168240
> Just curious. Why would you run `system.exit`? You should better just throw a proper exception. Calling `system.exit` inside the UDFs wouldn't terminate user applications as intended.
Below is the case I encountered, and it's cause by the code had a method named 'testUdf' and a udf obj also named 'testUdf', then worker exit. And I think the application should not hangup without any information anyway
```
import pyspark.sql.functions as F
from pyspark.sql.types import *
from pyspark.sql import SparkSession
def testUdf(x):
return x+1
spark = SparkSession.builder.appName("test").getOrCreate()
testUdf=F.udf(lambda x: testUdf(x),DoubleType())
df = spark.sql("select float(2.145025959889993) as a")
df.withColumn("test", testUdf(df.a)).show()
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-724619887
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wzhfy commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception
Posted by GitBox <gi...@apache.org>.
wzhfy commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-722260664
jenkins, test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-724552555
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-722286893
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35257/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-724603944
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35462/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-724605414
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35460/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] li36909 commented on a change in pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
li36909 commented on a change in pull request #30248:
URL: https://github.com/apache/spark/pull/30248#discussion_r520380016
##########
File path: python/pyspark/tests/test_worker.py
##########
@@ -95,6 +95,14 @@ def raise_exception(_):
self.assertRaises(Exception, lambda: rdd.foreach(raise_exception))
self.assertEqual(100, rdd.map(str).count())
+ def test_after_non_exception_error(self):
Review comment:
updated
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-724619887
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wzhfy commented on a change in pull request #30248: [SPARK-33339][PYTHON] Pyspark application maybe hangup because of wor…
Posted by GitBox <gi...@apache.org>.
wzhfy commented on a change in pull request #30248:
URL: https://github.com/apache/spark/pull/30248#discussion_r517839698
##########
File path: python/pyspark/tests/test_worker.py
##########
@@ -95,6 +95,14 @@ def raise_exception(_):
self.assertRaises(Exception, lambda: rdd.foreach(raise_exception))
self.assertEqual(100, rdd.map(str).count())
+ def test_after_workerExit(self):
Review comment:
`test_after_workerExit` -> `test_after_non_exception_error` ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-722280018
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-722296685
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application maybe hangup because of wor…
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-721686486
Just curious. Why would you run `system.exit`? You should better just throw a proper exception. Calling `system.exit` inside the UDFs wouldn't terminate user applications as intended.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-724618468
Merged to master and branch-3.0.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wzhfy edited a comment on pull request #30248: [SPARK-33339][PYTHON] Pyspark application maybe hangup because of wor…
Posted by GitBox <gi...@apache.org>.
wzhfy edited a comment on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-722185651
@li36909 Could you update the description? Currently it's misleading.
@HyukjinKwon Actually `system.exit` can happen when user writes some incorrect code (e.g. try to create a SparkContext in a executor task while no SPARK_HOME is available on that node, or the udf case @li36909 mentioned).
We have found several such cases from unskilled PySpark customers.
It would be better for Spark to throw an exception to the user, rather than hanging and making users not know what to do.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-722279481
**[Test build #130646 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130646/testReport)** for PR 30248 at commit [`fb8d325`](https://github.com/apache/spark/commit/fb8d325c7241373ae4d2d66bb0ed2e2a87ff3a17).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #30248:
URL: https://github.com/apache/spark/pull/30248#discussion_r520267220
##########
File path: python/pyspark/tests/test_worker.py
##########
@@ -95,6 +95,14 @@ def raise_exception(_):
self.assertRaises(Exception, lambda: rdd.foreach(raise_exception))
self.assertEqual(100, rdd.map(str).count())
+ def test_after_non_exception_error(self):
Review comment:
Can you add a short description with JIRA id? See "Pull Request" in http://spark.apache.org/contributing.html
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-722296685
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] li36909 commented on a change in pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
li36909 commented on a change in pull request #30248:
URL: https://github.com/apache/spark/pull/30248#discussion_r520379616
##########
File path: python/pyspark/tests/test_worker.py
##########
@@ -95,6 +95,14 @@ def raise_exception(_):
self.assertRaises(Exception, lambda: rdd.foreach(raise_exception))
self.assertEqual(100, rdd.map(str).count())
+ def test_after_non_exception_error(self):
+ def raise_SystemExit(_):
Review comment:
updated
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wzhfy edited a comment on pull request #30248: [SPARK-33339][PYTHON] Pyspark application maybe hangup because of wor…
Posted by GitBox <gi...@apache.org>.
wzhfy edited a comment on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-722185651
@li36909 Could you update the description? Currently it's misleading.
@HyukjinKwon Actually `system.exit` is throw because user writes some incorrect code (e.g. try to create a SparkContext in a executor task while no SPARK_HOME is available on that node, or the udf case @li36909 mentioned). We have found several such cases from unskilled Spark customers.
It would be better for Spark to throw an exception to the user, rather than hanging and making users not know what to do.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #30248:
URL: https://github.com/apache/spark/pull/30248#discussion_r520265487
##########
File path: python/pyspark/tests/test_worker.py
##########
@@ -95,6 +95,14 @@ def raise_exception(_):
self.assertRaises(Exception, lambda: rdd.foreach(raise_exception))
self.assertEqual(100, rdd.map(str).count())
+ def test_after_non_exception_error(self):
+ def raise_SystemExit(_):
Review comment:
`raise_SystemExit` -> `raise_system_exit`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] li36909 edited a comment on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception
Posted by GitBox <gi...@apache.org>.
li36909 edited a comment on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-722330855
@HyukjinKwon @wzhfy
does this seem good yo tou?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-722282653
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35256/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-724621159
It has a conflict in branch-2.4. It's sort of a corner case so I think we don't bother porting it back. @li36909, please go ahead and open a PR to backport if you're willing to do.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wzhfy commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception
Posted by GitBox <gi...@apache.org>.
wzhfy commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-722259634
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wzhfy commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception
Posted by GitBox <gi...@apache.org>.
wzhfy commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-722421783
LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-724552545
**[Test build #130852 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130852/testReport)** for PR 30248 at commit [`8863dbb`](https://github.com/apache/spark/commit/8863dbbeaf5a20e283e7ebd92c2c55bc3a950998).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-724619858
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35462/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] li36909 commented on a change in pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
li36909 commented on a change in pull request #30248:
URL: https://github.com/apache/spark/pull/30248#discussion_r520359601
##########
File path: python/pyspark/worker.py
##########
@@ -604,7 +604,7 @@ def process():
# reuse.
TaskContext._setTaskContext(None)
BarrierTaskContext._setTaskContext(None)
- except Exception:
+ except BaseException:
Review comment:
at 621 line seems no need to cache BaseException, because daemon.py will cache SystemExit at 75 line
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wzhfy edited a comment on pull request #30248: [SPARK-33339][PYTHON] Pyspark application maybe hangup because of wor…
Posted by GitBox <gi...@apache.org>.
wzhfy edited a comment on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-722185651
@li36909 Could you update the description? Currently it's misleading.
@HyukjinKwon Actually `system.exit` can happen when user writes some incorrect code (e.g. try to create a SparkContext in a executor task while no SPARK_HOME is available on that node, or the udf case @li36909 mentioned).
We have found several such cases from unskilled PySpark customers.
It would be better for Spark to throw an exception to the user, rather than hanging and making user not know what to do.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wzhfy removed a comment on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception
Posted by GitBox <gi...@apache.org>.
wzhfy removed a comment on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-722421783
LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application maybe hangup because of wor…
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-721678421
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-722303252
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35257/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application maybe hangup because of wor…
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-721681053
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-724552555
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wzhfy edited a comment on pull request #30248: [SPARK-33339][PYTHON] Pyspark application maybe hangup because of wor…
Posted by GitBox <gi...@apache.org>.
wzhfy edited a comment on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-722185651
@li36909 Could you update the description? Currently it's misleading.
@HyukjinKwon Actually `system.exit` is throw because user writes some incorrect code (e.g. try to create a SparkContext in a executor task while no SPARK_HOME is available on that node, or the udf case @li36909 mentioned).
We have found several such cases from unskilled Spark customers.
It would be better for Spark to throw an exception to the user, rather than hanging and making users not know what to do.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wzhfy commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception
Posted by GitBox <gi...@apache.org>.
wzhfy commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-722769018
I'm ok with the change, but since I'm not very familiar with pySpark, @HyukjinKwon could you take another look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #30248:
URL: https://github.com/apache/spark/pull/30248
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #30248:
URL: https://github.com/apache/spark/pull/30248#discussion_r520390108
##########
File path: python/pyspark/tests/test_worker.py
##########
@@ -95,6 +95,15 @@ def raise_exception(_):
self.assertRaises(Exception, lambda: rdd.foreach(raise_exception))
self.assertEqual(100, rdd.map(str).count())
+ def test_after_non_exception_error(self):
+ # SPARK-33339: Pyspark application will hang due to non Exception
+ def raise_system_exit(_):
+ raise SystemExit()
+ rdd = self.sc.parallelize(range(100), 1)
+ with QuietTest(self.sc):
+ self.assertRaises(Exception, lambda: rdd.foreach(raise_SystemExit))
Review comment:
```suggestion
self.assertRaises(Exception, lambda: rdd.foreach(raise_system_exit))
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wzhfy edited a comment on pull request #30248: [SPARK-33339][PYTHON] Pyspark application maybe hangup because of wor…
Posted by GitBox <gi...@apache.org>.
wzhfy edited a comment on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-722185651
@li36909 Could you update the description? Currently it's misleading.
@HyukjinKwon Actually `system.exit` is throw because user writes some incorrect code (e.g. try to create a SparkContext in a executor task while no SPARK_HOME is available on that node, or the udf case @li36909 mentioned).
We have found several such cases from unskilled PySpark customers.
It would be better for Spark to throw an exception to the user, rather than hanging and making users not know what to do.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-722303277
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-724616654
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35460/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30248: [SPARK-33339][PYTHON] Pyspark application maybe hangup because of wor…
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-721678421
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-721681053
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception error
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-724577514
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30248: [SPARK-33339][PYTHON] Pyspark application will hang due to non Exception
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30248:
URL: https://github.com/apache/spark/pull/30248#issuecomment-722296659
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35256/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org