You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/04/22 02:16:32 UTC
[GitHub] [spark] kotlovs opened a new pull request #32283: [SPARK-34674][CORE][K8S] Close SparkContext after the Main method has finished
kotlovs opened a new pull request #32283:
URL: https://github.com/apache/spark/pull/32283
### What changes were proposed in this pull request?
Close SparkContext after the Main method has finished, to allow SparkApplication on K8S to complete.
This is fixed version of [closed PR](https://github.com/apache/spark/pull/32081).
### Why are the changes needed?
if I don't call the method sparkContext.stop() explicitly, then a Spark driver process doesn't terminate even after its Main method has been completed. This behaviour is different from spark on yarn, where the manual sparkContext stopping is not required. It looks like, the problem is in using non-daemon threads, which prevent the driver jvm process from terminating.
So I have inserted code that closes sparkContext automatically.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Manually on the production AWS EKS environment in my company.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #32283: [SPARK-34674][CORE][K8S] Close SparkContext after the Main method has finished
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #32283:
URL: https://github.com/apache/spark/pull/32283#discussion_r670906664
##########
File path: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
##########
@@ -952,6 +952,15 @@ private[spark] class SparkSubmit extends Logging {
} catch {
case t: Throwable =>
throw findCause(t)
+ } finally {
+ if (!isShell(args.primaryResource) && !isSqlShell(args.mainClass) &&
Review comment:
+1 to making it k8s specific ...
Ideally, we should be using daemon threads for most of our background work ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #32283: [SPARK-34674][CORE][K8S] Close SparkContext after the Main method has finished
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #32283:
URL: https://github.com/apache/spark/pull/32283
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32283: [SPARK-34674][CORE][K8S] Close SparkContext after the Main method has finished
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32283:
URL: https://github.com/apache/spark/pull/32283#issuecomment-824505223
**[Test build #137764 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137764/testReport)** for PR 32283 at commit [`814e391`](https://github.com/apache/spark/commit/814e391a0f7df53c7a1ad96c895c80f5745e591e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32283: [SPARK-34674][CORE][K8S] Close SparkContext after the Main method has finished
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32283:
URL: https://github.com/apache/spark/pull/32283#issuecomment-824523394
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42292/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32283: [SPARK-34674][CORE][K8S] Close SparkContext after the Main method has finished
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #32283:
URL: https://github.com/apache/spark/pull/32283#discussion_r671664415
##########
File path: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
##########
@@ -952,6 +952,15 @@ private[spark] class SparkSubmit extends Logging {
} catch {
case t: Throwable =>
throw findCause(t)
+ } finally {
+ if (!isShell(args.primaryResource) && !isSqlShell(args.mainClass) &&
Review comment:
Here is the PR.
- https://github.com/apache/spark/pull/33403
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32283: [SPARK-34674][CORE][K8S] Close SparkContext after the Main method has finished
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32283:
URL: https://github.com/apache/spark/pull/32283#issuecomment-824505223
**[Test build #137764 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137764/testReport)** for PR 32283 at commit [`814e391`](https://github.com/apache/spark/commit/814e391a0f7df53c7a1ad96c895c80f5745e591e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32283: [SPARK-34674][CORE][K8S] Close SparkContext after the Main method has finished
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #32283:
URL: https://github.com/apache/spark/pull/32283#discussion_r671662963
##########
File path: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
##########
@@ -952,6 +952,15 @@ private[spark] class SparkSubmit extends Logging {
} catch {
case t: Throwable =>
throw findCause(t)
+ } finally {
+ if (!isShell(args.primaryResource) && !isSqlShell(args.mainClass) &&
Review comment:
Since SPARK-34674 was released already via Apache Spark 3.1.2, I will make another JIRA for this, @HyukjinKwon , @mridulm , @Ngone51 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32283: [SPARK-34674][CORE][K8S] Close SparkContext after the Main method has finished
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32283:
URL: https://github.com/apache/spark/pull/32283#issuecomment-824489031
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #32283: [SPARK-34674][CORE][K8S] Close SparkContext after the Main method has finished
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #32283:
URL: https://github.com/apache/spark/pull/32283#issuecomment-824500261
ok to test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32283: [SPARK-34674][CORE][K8S] Close SparkContext after the Main method has finished
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32283:
URL: https://github.com/apache/spark/pull/32283#issuecomment-824523383
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42292/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32283: [SPARK-34674][CORE][K8S] Close SparkContext after the Main method has finished
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32283:
URL: https://github.com/apache/spark/pull/32283#discussion_r670881700
##########
File path: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
##########
@@ -952,6 +952,15 @@ private[spark] class SparkSubmit extends Logging {
} catch {
case t: Throwable =>
throw findCause(t)
+ } finally {
+ if (!isShell(args.primaryResource) && !isSqlShell(args.mainClass) &&
Review comment:
Can we check if we're in K8s here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32283: [SPARK-34674][CORE][K8S] Close SparkContext after the Main method has finished
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32283:
URL: https://github.com/apache/spark/pull/32283#issuecomment-824565949
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137764/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #32283: [SPARK-34674][CORE][K8S] Close SparkContext after the Main method has finished
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #32283:
URL: https://github.com/apache/spark/pull/32283#discussion_r670921528
##########
File path: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
##########
@@ -952,6 +952,15 @@ private[spark] class SparkSubmit extends Logging {
} catch {
case t: Throwable =>
throw findCause(t)
+ } finally {
+ if (!isShell(args.primaryResource) && !isSqlShell(args.mainClass) &&
Review comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32283: [SPARK-34674][CORE][K8S] Close SparkContext after the Main method has finished
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32283:
URL: https://github.com/apache/spark/pull/32283#issuecomment-824565949
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137764/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #32283: [SPARK-34674][CORE][K8S] Close SparkContext after the Main method has finished
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #32283:
URL: https://github.com/apache/spark/pull/32283#issuecomment-824500307
Thank you, @kotlovs .
cc @mridulm , too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32283: [SPARK-34674][CORE][K8S] Close SparkContext after the Main method has finished
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32283:
URL: https://github.com/apache/spark/pull/32283#issuecomment-824561915
**[Test build #137764 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137764/testReport)** for PR 32283 at commit [`814e391`](https://github.com/apache/spark/commit/814e391a0f7df53c7a1ad96c895c80f5745e591e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #32283: [SPARK-34674][CORE][K8S] Close SparkContext after the Main method has finished
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #32283:
URL: https://github.com/apache/spark/pull/32283#issuecomment-824559969
Jenkins also passed Java/Scala/Python tests and currently running irrelevant R test.
Merged to master/3.1. Thank you for your patience and persistence, @kotlovs !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32283: [SPARK-34674][CORE][K8S] Close SparkContext after the Main method has finished
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32283:
URL: https://github.com/apache/spark/pull/32283#issuecomment-824523394
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42292/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32283: [SPARK-34674][CORE][K8S] Close SparkContext after the Main method has finished
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32283:
URL: https://github.com/apache/spark/pull/32283#issuecomment-824489031
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org