You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/09/01 09:47:18 UTC
[GitHub] [spark] AngersZhuuuu opened a new pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
AngersZhuuuu opened a new pull request #33873:
URL: https://github.com/apache/spark/pull/33873
### What changes were proposed in this pull request?
In current code for yarn client mode, even when use use `yarn application -kill` to kill the application, driver side still exit with code 0. This behavior make job scheduler can't know the job is not success. and user don't know too.
In this case we should exit program with a non 0 code.
### Why are the changes needed?
Make scheduler/user more clear about application's status
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-928084675
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48162/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-930214251
ping @tgravescs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909133453
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-927975010
**[Test build #143649 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143649/testReport)** for PR 33873 at commit [`34f9d1a`](https://github.com/apache/spark/commit/34f9d1ad28c7e9f3cfbae47be72349e45c855fe6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-928084675
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48162/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-926808448
**[Test build #143609 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143609/testReport)** for PR 33873 at commit [`4501dae`](https://github.com/apache/spark/commit/4501daeb0230deedee3fcb939f1d26b82653dc1d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-916862762
> I am not sure I follow the query.
> Assuming the query is how to use handler to address the current usecase - the exit handling would move to the handler ?
But it can't solve the problem that, if we need to exit(1) program when application is failed/killed in yarn-client mode.
The handler can be moved to callback, but the main problem in this pr is if we should exit(1) program when application master failed or killed in yarn-client mode
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909074674
ping @tgravescs @yaooqinn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909868323
> > I agree, we should not call `System.exit` - client mode is used in all sorts of interesting ways.
>
> So, how about make a option? user can use configuration to decide if exit? and default keep current?
Also, there are always use client mode spark-sql/spark-shell/pyspark and user forgot to close it. When we just use yarn command to kill application. In current code, it only will make SparkContext stop, won't exit the whole program. Also ping @Ngone51
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909823030
I agree, we should not call `System.exit` - client mode is used in all sorts of interesting ways.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #33873:
URL: https://github.com/apache/spark/pull/33873#discussion_r699213193
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
##########
@@ -122,6 +122,13 @@ private[spark] class YarnClientSchedulerBackend(
}
allowInterrupt = false
sc.stop()
+ state match {
+ case FinalApplicationStatus.FAILED | FinalApplicationStatus.KILLED =>
+ logWarning(s"ApplicationMaster finished with status ${state}, " +
+ s"SparkContext should exit with code 1.")
+ System.exit(1)
Review comment:
> If it comes from cli or api call, it may not be good to exit directly
You mean there may have situation, user's code call kill and just let sc stop?
Hmm， it's a confusing problem...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-927995700
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143649/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-927230473
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143631/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-912899291
**[Test build #142991 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142991/testReport)** for PR 33873 at commit [`752d359`](https://github.com/apache/spark/commit/752d3591462a8d9f2240be2ca210bd0a249eb621).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909133453
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-913307605
> A callback sounds find to me.
> @AngersZhuuuu doesn't a callback require user code changes, which he has said is hard for him?
Not so sure about what you mean, can be more clear?
@mridulm you mean add a call back in `sc.stop()` like
```
def stop(callBack: Option[() => Unit] = None)
```
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-927232712
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48143/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909891591
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142903/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-912899304
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142991/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cxzl25 commented on a change in pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
cxzl25 commented on a change in pull request #33873:
URL: https://github.com/apache/spark/pull/33873#discussion_r699202293
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
##########
@@ -122,6 +122,13 @@ private[spark] class YarnClientSchedulerBackend(
}
allowInterrupt = false
sc.stop()
+ state match {
+ case FinalApplicationStatus.FAILED | FinalApplicationStatus.KILLED =>
+ logWarning(s"ApplicationMaster finished with status ${state}, " +
+ s"SparkContext should exit with code 1.")
+ System.exit(1)
Review comment:
If it comes from cli or api call, it may not be good to exit directly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-910565422
What we should have is probably a callback for sc `stop` - which can be used for a variety of other usecases as well for cleanup in user apps.
For the specific usecase that @AngersZhuuuu has here, the callback can exit.
Thoughts @tgravescs, @Ngone51 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909828244
> I agree, we should not call `System.exit` - client mode is used in all sorts of interesting ways.
So, how about make a option? user can use configuration to decide if exit? and default keep current?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-912295210
Gentle ping @tgravescs @Ngone51
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] asfgit closed pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #33873:
URL: https://github.com/apache/spark/pull/33873
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-927241161
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48143/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909823030
I agree, we should not call `System.exit` - client mode is used in all sorts of interesting ways.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-926859311
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48121/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #33873:
URL: https://github.com/apache/spark/pull/33873#discussion_r716677904
##########
File path: docs/running-on-yarn.md
##########
@@ -441,6 +441,16 @@ To use a custom metrics.properties for the application master and executors, upd
</td>
<td>1.6.0</td>
</tr>
+<tr>
+ <td><code>spark.yarn.am.clientModeExitOnErro</code></td>
Review comment:
missing r on Error
##########
File path: docs/running-on-yarn.md
##########
@@ -441,6 +441,16 @@ To use a custom metrics.properties for the application master and executors, upd
</td>
<td>1.6.0</td>
</tr>
+<tr>
+ <td><code>spark.yarn.am.clientModeExitOnErro</code></td>
+ <td>false</td>
+ <td>
+ In yarn-client mode when this is true, if driver got application report with final status of KILLED or FAILED,
+ driver will stop corresponding SparkContext and exit program with code 1.
+ Note that if this is true can called from another application and it will terminate the parent application as well.
Review comment:
Note, if this is true and called from another application, it will terminate the parent application as well.
##########
File path: docs/running-on-yarn.md
##########
@@ -441,6 +441,16 @@ To use a custom metrics.properties for the application master and executors, upd
</td>
<td>1.6.0</td>
</tr>
+<tr>
+ <td><code>spark.yarn.am.clientModeExitOnErro</code></td>
+ <td>false</td>
+ <td>
+ In yarn-client mode when this is true, if driver got application report with final status of KILLED or FAILED,
Review comment:
add , after mode and before when
use:
"if the driver received an application report"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-926855733
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48121/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-926859311
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48121/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-930324842
merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-927995345
**[Test build #143649 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143649/testReport)** for PR 33873 at commit [`34f9d1a`](https://github.com/apache/spark/commit/34f9d1ad28c7e9f3cfbae47be72349e45c855fe6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-916596513
I am not sure I follow the query.
Assuming the query is how to use handler to address the current usecase - the exit handling would move to the handler ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909093081
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909111309
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142880/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #33873:
URL: https://github.com/apache/spark/pull/33873#discussion_r699213193
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
##########
@@ -122,6 +122,13 @@ private[spark] class YarnClientSchedulerBackend(
}
allowInterrupt = false
sc.stop()
+ state match {
+ case FinalApplicationStatus.FAILED | FinalApplicationStatus.KILLED =>
+ logWarning(s"ApplicationMaster finished with status ${state}, " +
+ s"SparkContext should exit with code 1.")
+ System.exit(1)
Review comment:
> If it comes from cli or api call, it may not be good to exit directly
You mean there may have situation, user's code call kill and just let sc stop?
Hmm， it's a confusing problem...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #33873:
URL: https://github.com/apache/spark/pull/33873#discussion_r699213193
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
##########
@@ -122,6 +122,13 @@ private[spark] class YarnClientSchedulerBackend(
}
allowInterrupt = false
sc.stop()
+ state match {
+ case FinalApplicationStatus.FAILED | FinalApplicationStatus.KILLED =>
+ logWarning(s"ApplicationMaster finished with status ${state}, " +
+ s"SparkContext should exit with code 1.")
+ System.exit(1)
Review comment:
> If it comes from cli or api call, it may not be good to exit directly
You mean there may have situation, user's code call kill and just let sc stop?
Hmm， it's a confusing problem...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #33873:
URL: https://github.com/apache/spark/pull/33873#discussion_r716776094
##########
File path: docs/running-on-yarn.md
##########
@@ -441,6 +441,16 @@ To use a custom metrics.properties for the application master and executors, upd
</td>
<td>1.6.0</td>
</tr>
+<tr>
+ <td><code>spark.yarn.am.clientModeExitOnErro</code></td>
+ <td>false</td>
+ <td>
+ In yarn-client mode when this is true, if driver got application report with final status of KILLED or FAILED,
+ driver will stop corresponding SparkContext and exit program with code 1.
+ Note that if this is true can called from another application and it will terminate the parent application as well.
Review comment:
Done
##########
File path: docs/running-on-yarn.md
##########
@@ -441,6 +441,16 @@ To use a custom metrics.properties for the application master and executors, upd
</td>
<td>1.6.0</td>
</tr>
+<tr>
+ <td><code>spark.yarn.am.clientModeExitOnErro</code></td>
+ <td>false</td>
+ <td>
+ In yarn-client mode when this is true, if driver got application report with final status of KILLED or FAILED,
Review comment:
Done
##########
File path: docs/running-on-yarn.md
##########
@@ -441,6 +441,16 @@ To use a custom metrics.properties for the application master and executors, upd
</td>
<td>1.6.0</td>
</tr>
+<tr>
+ <td><code>spark.yarn.am.clientModeExitOnErro</code></td>
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-930324842
merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-912899197
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47493/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-927995700
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143649/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909093081
**[Test build #142880 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142880/testReport)** for PR 33873 at commit [`ff019fd`](https://github.com/apache/spark/commit/ff019fd6411583d4a3d28f7e731a2062a015d0a5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-912904455
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47494/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cxzl25 commented on a change in pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
cxzl25 commented on a change in pull request #33873:
URL: https://github.com/apache/spark/pull/33873#discussion_r699202293
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
##########
@@ -122,6 +122,13 @@ private[spark] class YarnClientSchedulerBackend(
}
allowInterrupt = false
sc.stop()
+ state match {
+ case FinalApplicationStatus.FAILED | FinalApplicationStatus.KILLED =>
+ logWarning(s"ApplicationMaster finished with status ${state}, " +
+ s"SparkContext should exit with code 1.")
+ System.exit(1)
Review comment:
If it comes from cli or api call, it may not be good to exit directly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-912904452
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47494/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909242265
this comes back to the same thing we discussed before (https://github.com/apache/spark/pull/33780). Client mode always returns success right now. It seems you are trying to change that here in a slightly different way then the other PR.
https://issues.apache.org/jira/browse/SPARK-3627 has some history on that.
Note that here is where they don't want system.exit to happen in client mode: https://issues.apache.org/jira/browse/SPARK-1516
I think if you want to change the behavior of yarn client mode please start a mail on spark dev to get feedback, I'm not sure who might all be interested. Adding a few here @mirdulm @dbtsai @srowen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-914655782
By callback I meant some handler (when provided) which is invoked when a state transition completes.
A few examples would be:
* When spark context has completed initialization - last thing initialization does.
* When spark context has completed shutting down - last thing sc.stop does.
This will allow for custom initialization/termination logic to be added without modifying spark in order to support some specific usecase.
The default behavior of spark would continue, and users can customize/override how their spark applications behave.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-927240287
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48143/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-928072299
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48162/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-930214251
ping @tgravescs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909111309
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909892065
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47406/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909111309
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909105007
**[Test build #142882 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142882/testReport)** for PR 33873 at commit [`1553602`](https://github.com/apache/spark/commit/155360290d64757b09613bfa92fa0aac8d07227a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-912899600
**[Test build #142992 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142992/testReport)** for PR 33873 at commit [`94021f5`](https://github.com/apache/spark/commit/94021f5c53ac7a80b5a018ee62ea98ab65993bb0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909105007
**[Test build #142882 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142882/testReport)** for PR 33873 at commit [`1553602`](https://github.com/apache/spark/commit/155360290d64757b09613bfa92fa0aac8d07227a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] holdenk commented on a change in pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
holdenk commented on a change in pull request #33873:
URL: https://github.com/apache/spark/pull/33873#discussion_r707607075
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/config.scala
##########
@@ -52,6 +52,14 @@ package object config extends Logging {
.timeConf(TimeUnit.MILLISECONDS)
.createOptional
+ private[spark] val AM_CLIENT_MODE_EXIT_DIRECTLY =
+ ConfigBuilder("spark.yarn.am.clientModeExitDirectly")
+ .doc("When ture, if YarnClientSchedulerBackend.MonitorThread got report with " +
Review comment:
nit: s/got/gets/
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
##########
@@ -122,6 +122,13 @@ private[spark] class YarnClientSchedulerBackend(
}
allowInterrupt = false
sc.stop()
+ state match {
+ case FinalApplicationStatus.FAILED | FinalApplicationStatus.KILLED =>
+ logWarning(s"ApplicationMaster finished with status ${state}, " +
+ s"SparkContext should exit with code 1.")
+ System.exit(1)
Review comment:
yeah ok so we could be in the situation where someone is using Spark as part of another application (e.g. some analytics). I think a good compromise for now could be adding another flag for "exit on driver error" or just switching this to a logError and dropping `system.exit`. What do folks think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-928036506
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48162/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-926820014
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48121/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-926793384
**[Test build #143609 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143609/testReport)** for PR 33873 at commit [`4501dae`](https://github.com/apache/spark/commit/4501daeb0230deedee3fcb939f1d26b82653dc1d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-926808692
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143609/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-926793384
**[Test build #143609 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143609/testReport)** for PR 33873 at commit [`4501dae`](https://github.com/apache/spark/commit/4501daeb0230deedee3fcb939f1d26b82653dc1d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909892032
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47406/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909093081
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #33873:
URL: https://github.com/apache/spark/pull/33873#discussion_r707878507
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
##########
@@ -122,6 +122,13 @@ private[spark] class YarnClientSchedulerBackend(
}
allowInterrupt = false
sc.stop()
+ state match {
+ case FinalApplicationStatus.FAILED | FinalApplicationStatus.KILLED =>
+ logWarning(s"ApplicationMaster finished with status ${state}, " +
+ s"SparkContext should exit with code 1.")
+ System.exit(1)
Review comment:
> yeah ok so we could be in the situation where someone is using Spark as part of another application (e.g. some analytics). I think a good compromise for now could be adding another flag for "exit on driver error" or just switching this to a logError and dropping `system.exit`. What do folks think?
You mean change the configuration to `exit on driver error` and default is true, when false, if application master kill/failed, driver exit too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-921846421
> yeah exit on error seems ok. its long the same lines of https://github.com/apache/spark/pull/33780/files.
Hmmm, not understand what do you mean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-921806070
yeah exit on error seems ok. its long the same lines of https://github.com/apache/spark/pull/33780/files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909868267
**[Test build #142903 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142903/testReport)** for PR 33873 at commit [`c27c9ef`](https://github.com/apache/spark/commit/c27c9ef36cdcdd617e2eac93bba4e7f25f46399c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909879731
**[Test build #142903 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142903/testReport)** for PR 33873 at commit [`c27c9ef`](https://github.com/apache/spark/commit/c27c9ef36cdcdd617e2eac93bba4e7f25f46399c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909121567
**[Test build #142882 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142882/testReport)** for PR 33873 at commit [`1553602`](https://github.com/apache/spark/commit/155360290d64757b09613bfa92fa0aac8d07227a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909868267
**[Test build #142903 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142903/testReport)** for PR 33873 at commit [`c27c9ef`](https://github.com/apache/spark/commit/c27c9ef36cdcdd617e2eac93bba4e7f25f46399c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-915743493
> By callback I meant some handler (when provided) which is invoked when a state transition completes.
> A few examples could be:
>
> * When spark context has completed initialization - last thing initialization does.
> * When spark context has completed shutting down - last thing sc.stop does.
>
> This will allow for custom initialization/termination logic to be added without modifying spark in order to support some specific usecase.
> The default behavior of spark would continue, and users can customize/override how their spark applications behave.
Got your point, but how it can be used in this case? Or can we solve this first then discuss on sparkcontext handler API?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-912904455
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47494/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-927975010
**[Test build #143649 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143649/testReport)** for PR 33873 at commit [`34f9d1a`](https://github.com/apache/spark/commit/34f9d1ad28c7e9f3cfbae47be72349e45c855fe6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-927228151
**[Test build #143631 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143631/testReport)** for PR 33873 at commit [`fc7c271`](https://github.com/apache/spark/commit/fc7c2716e6231982f6bae91e30e6a2aac5e27aa2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-912895370
> A callback sounds find to me.
> @AngersZhuuuu doesn't a callback require user code changes, which he has said is hard for him?
Not so sure about what you mean, you mean user feel hard to write a call back?
Add a callback can solve some case, you mean we can add a callback in YarnClientSchedulerBackend.MonitorThread's `sc.stop()`.
Such as add a Shutdown hook to the callback, if the program ended and the status is failed and killed, we call this shutdown hook to exit with code 1? @mridulm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-912899199
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47493/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-912540068
A callback sounds find to me.
@AngersZhuuuu doesn't a callback require user code changes, which he has said is hard for him?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-912903640
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47494/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-912899304
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142991/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909173736
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47385/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909136509
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47385/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909093081
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-912901726
**[Test build #142992 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142992/testReport)** for PR 33873 at commit [`94021f5`](https://github.com/apache/spark/commit/94021f5c53ac7a80b5a018ee62ea98ab65993bb0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-926808692
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143609/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm edited a comment on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
mridulm edited a comment on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-910565422
One possibility is to add a callback for sc `stop` - which can be used for a variety of other usecases, including for cleanup in user apps.
For the specific usecase that @AngersZhuuuu has here, the callback can exit.
Thoughts @tgravescs, @Ngone51 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-910338385
> Can't users use the yarn command, e.g., `yarn application -status application_1459542433815_0002`, to know the status?
But so many job, user can't check all jobs. IMO, add a configuration for different case to handle this can be ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-928084675
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48162/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909242265
this comes back to the same thing we discussed before (https://github.com/apache/spark/pull/33780). Client mode always returns success right now. It seems you are trying to change that here in a slightly different way then the other PR.
https://issues.apache.org/jira/browse/SPARK-3627 has some history on that.
Note that here is where they don't want system.exit to happen in client mode: https://issues.apache.org/jira/browse/SPARK-1516
I think if you want to change the behavior of yarn client mode please start a mail on spark dev to get feedback, I'm not sure who might all be interested. Adding a few here @mirdulm @dbtsai @srowen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909093081
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909173736
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47385/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909126740
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47383/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu removed a comment on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu removed a comment on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-912895370
> A callback sounds find to me.
> @AngersZhuuuu doesn't a callback require user code changes, which he has said is hard for him?
Not so sure about what you mean, you mean user feel hard to write a call back?
Add a callback can solve some case, you mean we can add a callback in YarnClientSchedulerBackend.MonitorThread's `sc.stop()`.
Such as add a Shutdown hook to the callback, if the program ended and the status is failed and killed, we call this shutdown hook to exit with code 1? @mridulm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909164398
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47385/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu removed a comment on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu removed a comment on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-921846421
> yeah exit on error seems ok. its long the same lines of https://github.com/apache/spark/pull/33780/files.
Hmmm, not understand what do you mean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909074674
ping @tgravescs @yaooqinn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #33873:
URL: https://github.com/apache/spark/pull/33873#discussion_r715763532
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/config.scala
##########
@@ -52,6 +52,14 @@ package object config extends Logging {
.timeConf(TimeUnit.MILLISECONDS)
.createOptional
+ private[spark] val AM_CLIENT_MODE_EXIT_DIRECTLY =
+ ConfigBuilder("spark.yarn.am.clientModeExitDirectly")
+ .doc("When ture, if YarnClientSchedulerBackend.MonitorThread got report with " +
Review comment:
> true is misspelled, perhaps rename to ClientModeExitOnError, I don't think user case about MonitorThread, I think we should state this more that it will system exit, but note that if this is true can called from another application it will terminate the parent application as well.
Nice suggestion, Your consideration is very comprehensive， updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909891591
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-926297580
> yeah exit on error seems ok. its long the same lines of https://github.com/apache/spark/pull/33780/files.
Hmmm, sorry, I am not clear what do you mean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-927230429
**[Test build #143631 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143631/testReport)** for PR 33873 at commit [`fc7c271`](https://github.com/apache/spark/commit/fc7c2716e6231982f6bae91e30e6a2aac5e27aa2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] asfgit closed pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #33873:
URL: https://github.com/apache/spark/pull/33873
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-927228151
**[Test build #143631 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143631/testReport)** for PR 33873 at commit [`fc7c271`](https://github.com/apache/spark/commit/fc7c2716e6231982f6bae91e30e6a2aac5e27aa2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-927241161
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48143/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909111309
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-912898560
**[Test build #142991 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142991/testReport)** for PR 33873 at commit [`752d359`](https://github.com/apache/spark/commit/752d3591462a8d9f2240be2ca210bd0a249eb621).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm edited a comment on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
mridulm edited a comment on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-914655782
By callback I meant some handler (when provided) which is invoked when a state transition completes.
A few examples could be:
* When spark context has completed initialization - last thing initialization does.
* When spark context has completed shutting down - last thing sc.stop does.
This will allow for custom initialization/termination logic to be added without modifying spark in order to support some specific usecase.
The default behavior of spark would continue, and users can customize/override how their spark applications behave.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-912903946
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142992/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909074674
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909242265
this comes back to the same thing we discussed before (https://github.com/apache/spark/pull/33780). Client mode always returns success right now. It seems you are trying to change that here in a slightly different way then the other PR.
https://issues.apache.org/jira/browse/SPARK-3627 has some history on that.
Note that here is where they don't want system.exit to happen in client mode: https://issues.apache.org/jira/browse/SPARK-1516
I think if you want to change the behavior of yarn client mode please start a mail on spark dev to get feedback, I'm not sure who might all be interested. Adding a few here @mirdulm @dbtsai @srowen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909125414
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47383/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-912899600
**[Test build #142992 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142992/testReport)** for PR 33873 at commit [`94021f5`](https://github.com/apache/spark/commit/94021f5c53ac7a80b5a018ee62ea98ab65993bb0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909111309
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142880/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909111309
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #33873:
URL: https://github.com/apache/spark/pull/33873#discussion_r715639393
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/config.scala
##########
@@ -52,6 +52,14 @@ package object config extends Logging {
.timeConf(TimeUnit.MILLISECONDS)
.createOptional
+ private[spark] val AM_CLIENT_MODE_EXIT_DIRECTLY =
+ ConfigBuilder("spark.yarn.am.clientModeExitDirectly")
+ .doc("When ture, if YarnClientSchedulerBackend.MonitorThread got report with " +
Review comment:
true is misspelled, perhaps rename to ClientModeExitOnError, I don't think user case about MonitorThread, I think we should state this more that it will system exit, but note that if this is true can called from another application it will terminate the parent application as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-927230473
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143631/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-912899199
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47493/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-912898560
**[Test build #142991 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142991/testReport)** for PR 33873 at commit [`752d359`](https://github.com/apache/spark/commit/752d3591462a8d9f2240be2ca210bd0a249eb621).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909111031
**[Test build #142880 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142880/testReport)** for PR 33873 at commit [`ff019fd`](https://github.com/apache/spark/commit/ff019fd6411583d4a3d28f7e731a2062a015d0a5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-909093081
**[Test build #142880 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142880/testReport)** for PR 33873 at commit [`ff019fd`](https://github.com/apache/spark/commit/ff019fd6411583d4a3d28f7e731a2062a015d0a5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cxzl25 commented on a change in pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
cxzl25 commented on a change in pull request #33873:
URL: https://github.com/apache/spark/pull/33873#discussion_r699202293
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
##########
@@ -122,6 +122,13 @@ private[spark] class YarnClientSchedulerBackend(
}
allowInterrupt = false
sc.stop()
+ state match {
+ case FinalApplicationStatus.FAILED | FinalApplicationStatus.KILLED =>
+ logWarning(s"ApplicationMaster finished with status ${state}, " +
+ s"SparkContext should exit with code 1.")
+ System.exit(1)
Review comment:
If it comes from cli or api call, it may not be good to exit directly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33873: [SPARK-36624][YARN] In yarn client mode, when ApplicationMaster failed with KILLED/FAILED, driver should exit with code not 0
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33873:
URL: https://github.com/apache/spark/pull/33873#issuecomment-912903946
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142992/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org