You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/08/18 08:37:35 UTC
[GitHub] [spark] AngersZhuuuu opened a new pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
AngersZhuuuu opened a new pull request #33780:
URL: https://github.com/apache/spark/pull/33780
### What changes were proposed in this pull request?
We meet a case AM lose connection
```
21/08/18 02:14:15 ERROR TransportRequestHandler: Error sending result RpcResponse{requestId=5675952834716124039, body=NioManagedBuffer{buf=java.nio.HeapByteBuffer[pos=0 lim=47 cap=64]}} to xx.xx.xx.xx:41420; closing connection
java.nio.channels.ClosedChannelException
at io.netty.channel.AbstractChannel$AbstractUnsafe.newClosedChannelException(AbstractChannel.java:957)
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:865)
at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1367)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)
at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:764)
at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1104)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
```
Check the code about client, when AMEndpoint disconnected, will finish Application with SUCCESS final status
```
override def onDisconnected(remoteAddress: RpcAddress): Unit = {
// In cluster mode or unmanaged am case, do not rely on the disassociated event to exit
// This avoids potentially reporting incorrect exit codes if the driver fails
if (!(isClusterMode || sparkConf.get(YARN_UNMANAGED_AM))) {
logInfo(s"Driver terminated or disconnected! Shutting down. $remoteAddress")
finish(FinalApplicationStatus.SUCCEEDED, ApplicationMaster.EXIT_SUCCESS)
}
}
```
Normally say in client mode, when application success, driver will stop and AM loss connection, it's ok that exit with SUCCESS, but if there is a not work problem cause disconnected. Still finish with final status is not correct.
Then YarnClientSchedulerBackend will receive application report with final status with success and stop SparkContext cause application failed but mark it as a normal stop.
```
private class MonitorThread extends Thread {
private var allowInterrupt = true
override def run() {
try {
val YarnAppReport(_, state, diags) =
client.monitorApplication(appId.get, logApplicationReport = false)
logError(s"YARN application has exited unexpectedly with state $state! " +
"Check the YARN application logs for more details.")
diags.foreach { err =>
logError(s"Diagnostics message: $err")
}
allowInterrupt = false
sc.stop()
} catch {
case e: InterruptedException => logInfo("Interrupting monitor thread")
}
}
def stopMonitor(): Unit = {
if (allowInterrupt) {
this.interrupt()
}
```
IMO, we should send a `Shutdown` message to yarn client mode AM to make sure the shut down case
### Why are the changes needed?
Fix bug
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-915739860
> @tgravescs has much more context on this, I will defer to his advice on this.
Ok, seem no other dev care about this. Add a configuration is the only I can find that meet different case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906553616
**[Test build #142820 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142820/testReport)** for PR 33780 at commit [`af00e87`](https://github.com/apache/spark/commit/af00e8747987cb6b950d60aabd713082491f4408).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906143492
**[Test build #142791 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142791/testReport)** for PR 33780 at commit [`db3b64e`](https://github.com/apache/spark/commit/db3b64efcf93ee582a36b7dd034c589c7ba2ba36).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-903402263
Also ping @vanzin @xkrogen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-909315224
> Like mentioned in the other issue, I think we should start discussion on spark dev..
The email has been sent, if I missed anything, please add it. Thanks a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906144355
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142791/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-900966484
**[Test build #142603 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142603/testReport)** for PR 33780 at commit [`2d9578c`](https://github.com/apache/spark/commit/2d9578ca92de3b459c34833847de47e7664de61b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-900938276
**[Test build #142598 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142598/testReport)** for PR 33780 at commit [`7cdffa9`](https://github.com/apache/spark/commit/7cdffa9d7420de424bce22b8f1504deedd5dddec).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-932278578
**[Test build #143786 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143786/testReport)** for PR 33780 at commit [`26ad1ca`](https://github.com/apache/spark/commit/26ad1ca4da8d50b0c6f40c0cc1609d6c23231c39).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-909090852
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906128069
ping @yaooqinn Can you take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-900973331
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142603/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906177581
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-900938364
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906152049
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142793/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-901061343
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47108/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-938676516
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48501/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-938787713
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48505/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-938847931
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48505/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-909053152
**[Test build #142878 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142878/testReport)** for PR 33780 at commit [`d90816e`](https://github.com/apache/spark/commit/d90816e4798ad0ada5af2a7f5e60a405e6b8e98d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-909258399
Like mentioned in the other issue, I think we should start discussion on spark dev..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-909258399
Like mentioned in the other issue, I think we should start discussion on spark dev..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-900938364
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-930356264
**[Test build #143720 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143720/testReport)** for PR 33780 at commit [`a0d4b81`](https://github.com/apache/spark/commit/a0d4b8173ea0c2a93e613e419e3be01f596ab32d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-930356644
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143720/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-909053152
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906568454
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47324/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906607446
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47324/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906531655
> If we were to do this it would definitely need more explanation in the config settings.
update the desc doc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-901031186
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47108/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-930336716
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] asfgit closed pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #33780:
URL: https://github.com/apache/spark/pull/33780
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-938722045
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48501/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-938661890
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144024/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-938705690
**[Test build #144026 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144026/testReport)** for PR 33780 at commit [`9e14c21`](https://github.com/apache/spark/commit/9e14c21f8e2047ddbd4b8975d2fa5c71b097eff0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #33780:
URL: https://github.com/apache/spark/pull/33780#discussion_r699136313
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -793,6 +796,8 @@ private[spark] class ApplicationMaster(
override def receive: PartialFunction[Any, Unit] = {
case UpdateDelegationTokens(tokens) =>
SparkHadoopUtil.get.addDelegationTokens(tokens, sparkConf)
+
+ case Shutdown => shutdown = true
Review comment:
@tgravescs move Shutdown to `receive` since it's a OneWayMessage.
And have tested this pr in our prod by use gdb to close socket between driver and am.
The behavior is as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-901035918
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47104/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-900982416
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906170561
**[Test build #142794 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142794/testReport)** for PR 33780 at commit [`07d42c5`](https://github.com/apache/spark/commit/07d42c5d61a3573579c163e59c5d49186e7573f4).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #33780:
URL: https://github.com/apache/spark/pull/33780#discussion_r699136313
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -793,6 +796,8 @@ private[spark] class ApplicationMaster(
override def receive: PartialFunction[Any, Unit] = {
case UpdateDelegationTokens(tokens) =>
SparkHadoopUtil.get.addDelegationTokens(tokens, sparkConf)
+
+ case Shutdown => shutdown = true
Review comment:
@tgravescs move Shutdown to `receive` since it's a OneWayMessage.
And have tested this pr in our prod by use gdb to close socket between driver and am.
The behavior is as expected.
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -793,6 +796,8 @@ private[spark] class ApplicationMaster(
override def receive: PartialFunction[Any, Unit] = {
case UpdateDelegationTokens(tokens) =>
SparkHadoopUtil.get.addDelegationTokens(tokens, sparkConf)
+
+ case Shutdown => shutdown = true
Review comment:
FYI @tgravescs move Shutdown to `receive` since it's a OneWayMessage.
And have tested this pr in our prod by use gdb to close socket between driver and am.
The behavior is as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-909053152
**[Test build #142878 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142878/testReport)** for PR 33780 at commit [`d90816e`](https://github.com/apache/spark/commit/d90816e4798ad0ada5af2a7f5e60a405e6b8e98d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906148930
**[Test build #142793 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142793/testReport)** for PR 33780 at commit [`07d42c5`](https://github.com/apache/spark/commit/07d42c5d61a3573579c163e59c5d49186e7573f4).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-901646930
cc @tgravescs @mridulm FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906144339
**[Test build #142791 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142791/testReport)** for PR 33780 at commit [`db3b64e`](https://github.com/apache/spark/commit/db3b64efcf93ee582a36b7dd034c589c7ba2ba36).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906553900
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142820/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #33780:
URL: https://github.com/apache/spark/pull/33780
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-938748955
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-930336716
**[Test build #143720 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143720/testReport)** for PR 33780 at commit [`a0d4b81`](https://github.com/apache/spark/commit/a0d4b8173ea0c2a93e613e419e3be01f596ab32d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-932298081
**[Test build #143786 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143786/testReport)** for PR 33780 at commit [`26ad1ca`](https://github.com/apache/spark/commit/26ad1ca4da8d50b0c6f40c0cc1609d6c23231c39).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-930356644
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-909133464
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47381/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-915430682
@tgravescs has much more context on this, I will defer to his advice on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906152009
**[Test build #142793 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142793/testReport)** for PR 33780 at commit [`07d42c5`](https://github.com/apache/spark/commit/07d42c5d61a3573579c163e59c5d49186e7573f4).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #33780:
URL: https://github.com/apache/spark/pull/33780#discussion_r699136313
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -793,6 +796,8 @@ private[spark] class ApplicationMaster(
override def receive: PartialFunction[Any, Unit] = {
case UpdateDelegationTokens(tokens) =>
SparkHadoopUtil.get.addDelegationTokens(tokens, sparkConf)
+
+ case Shutdown => shutdown = true
Review comment:
@tgravescs move Shutdown to `receive` since it's a OneWayMessage.
And have tested this pr in our prod by use gdb to close socket between driver and am.
The behavior is as expected.
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -793,6 +796,8 @@ private[spark] class ApplicationMaster(
override def receive: PartialFunction[Any, Unit] = {
case UpdateDelegationTokens(tokens) =>
SparkHadoopUtil.get.addDelegationTokens(tokens, sparkConf)
+
+ case Shutdown => shutdown = true
Review comment:
FYI @tgravescs move Shutdown to `receive` since it's a OneWayMessage.
And have tested this pr in our prod by use gdb to close socket between driver and am.
The behavior is as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906183567
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47294/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-900966484
**[Test build #142603 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142603/testReport)** for PR 33780 at commit [`2d9578c`](https://github.com/apache/spark/commit/2d9578ca92de3b459c34833847de47e7664de61b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-900988813
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47098/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906152569
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-909119141
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47381/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906143492
**[Test build #142791 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142791/testReport)** for PR 33780 at commit [`db3b64e`](https://github.com/apache/spark/commit/db3b64efcf93ee582a36b7dd034c589c7ba2ba36).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-938748958
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #33780:
URL: https://github.com/apache/spark/pull/33780#discussion_r724984806
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -784,6 +784,9 @@ private[spark] class ApplicationMaster(
*/
private class AMEndpoint(override val rpcEnv: RpcEnv, driver: RpcEndpointRef)
extends RpcEndpoint with Logging {
+ private var shutdown = false
Review comment:
this was not done, please update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-901037253
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906155470
**[Test build #142794 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142794/testReport)** for PR 33780 at commit [`07d42c5`](https://github.com/apache/spark/commit/07d42c5d61a3573579c163e59c5d49186e7573f4).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906176350
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47292/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-909053152
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-938694445
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-938742608
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48505/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906048695
ping @tgravescs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906183536
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47294/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-930336716
**[Test build #143720 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143720/testReport)** for PR 33780 at commit [`a0d4b81`](https://github.com/apache/spark/commit/a0d4b8173ea0c2a93e613e419e3be01f596ab32d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-901072565
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47108/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-901019677
**[Test build #142608 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142608/testReport)** for PR 33780 at commit [`dd56ddd`](https://github.com/apache/spark/commit/dd56dddfcf5d3aeab4a8e742d05f2fce3b187d20).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` protected class YarnSchedulerEndpoint(override val rpcEnv: RpcEnv)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-930356644
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #33780:
URL: https://github.com/apache/spark/pull/33780#discussion_r699136313
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -793,6 +796,8 @@ private[spark] class ApplicationMaster(
override def receive: PartialFunction[Any, Unit] = {
case UpdateDelegationTokens(tokens) =>
SparkHadoopUtil.get.addDelegationTokens(tokens, sparkConf)
+
+ case Shutdown => shutdown = true
Review comment:
FYI @tgravescs move Shutdown to `receive` since it's a OneWayMessage.
And have tested this pr in our prod by use gdb to close socket between driver and am.
The behavior is as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-901029633
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47106/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906532087
> I'm still hesitant on this, one I don't like more configs, two it seems like it could still give you the wrong result. I understand the use case though. I want to go back through some things to investigate a bit more.
Hope your more suggestion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-916904682
sorry I'm swamped right now, I'll get back to this next week to review and will try to ping a few people directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #33780:
URL: https://github.com/apache/spark/pull/33780#discussion_r720266715
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -784,6 +784,9 @@ private[spark] class ApplicationMaster(
*/
private class AMEndpoint(override val rpcEnv: RpcEnv, driver: RpcEndpointRef)
extends RpcEndpoint with Logging {
+ private var shutdown = false
Review comment:
> make shutdown @volatile
Done
##########
File path: docs/running-on-yarn.md
##########
@@ -441,6 +441,20 @@ To use a custom metrics.properties for the application master and executors, upd
</td>
<td>1.6.0</td>
</tr>
+<tr>
+ <td><code>spark.yarn.am.clientModeTreatDisconnectAsFailed</code></td>
+ <td>false</td>
+ <td>
+ In managed yarn-client mode, when am disconnect with driver, am will finish application with SUCCESS final status since in
Review comment:
DOne
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -843,8 +848,13 @@ private[spark] class ApplicationMaster(
// In cluster mode or unmanaged am case, do not rely on the disassociated event to exit
// This avoids potentially reporting incorrect exit codes if the driver fails
if (!(isClusterMode || sparkConf.get(YARN_UNMANAGED_AM))) {
- logInfo(s"Driver terminated or disconnected! Shutting down. $remoteAddress")
- finish(FinalApplicationStatus.SUCCEEDED, ApplicationMaster.EXIT_SUCCESS)
+ if (shutdown || !clientModeTreatDisconnectAsFailed) {
+ logInfo(s"Driver terminated or disconnected! Shutting down. $remoteAddress")
+ finish(FinalApplicationStatus.SUCCEEDED, ApplicationMaster.EXIT_SUCCESS)
+ } else {
+ logError(s"Application Master lose connection with driver! Shutting down. $remoteAddress")
Review comment:
DOne
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-930408751
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48232/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-938731258
**[Test build #144029 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144029/testReport)** for PR 33780 at commit [`9e14c21`](https://github.com/apache/spark/commit/9e14c21f8e2047ddbd4b8975d2fa5c71b097eff0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-932374077
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48298/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-901004448
**[Test build #142608 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142608/testReport)** for PR 33780 at commit [`dd56ddd`](https://github.com/apache/spark/commit/dd56dddfcf5d3aeab4a8e742d05f2fce3b187d20).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-900938364
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142598/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-900930760
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906518912
> Have you tried using the unmanagedAM? I don't have much experience with it but should resolve this because AM no longer remote.
Never tried this. Seems can avoid the problem about miss connection between driver and am, but it's really a huge usage change to all user..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-903833885
> The problem is that not everyone calls sc.stop... in interactive environments you also have people killing it weirdly, there were a bunch of shut down cases in client mode that made determining the real success/failure an issue.
>
> It was intentionally decided to just always return success in those cases. If this properly handles those cases I'm fine with it but I need to go try to remember what they all were and make sure they are handled. We should not report FAIL unless its truly a failure.
Got your point..I miss the case such as killing spark-shell weirdly...then how about:
- add a configuration such as spark.yarn.clientModeAMDisconnect as Failed, when true, make it as failed, when false let it behavior as before? then let user to choose? Two company's scheduler platform meet such case, so we can have a choose. Since we always non't need to rerun pysaprk/spark-shell/spark-sql?
- For pyspark/spark-shell/spark-sql such interactive mode, make it behavior as before and for normal batch job, make it failed when disconnected?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-932315765
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48298/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-932278578
**[Test build #143786 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143786/testReport)** for PR 33780 at commit [`26ad1ca`](https://github.com/apache/spark/commit/26ad1ca4da8d50b0c6f40c0cc1609d6c23231c39).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-930336716
**[Test build #143720 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143720/testReport)** for PR 33780 at commit [`a0d4b81`](https://github.com/apache/spark/commit/a0d4b8173ea0c2a93e613e419e3be01f596ab32d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-930366650
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48232/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-916919116
> sorry I'm swamped right now, I'll get back to this next week to review and will try to ping a few people directly.
Ok, thanks a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906617683
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47324/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-901048935
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47106/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-909090852
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906148930
**[Test build #142793 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142793/testReport)** for PR 33780 at commit [`07d42c5`](https://github.com/apache/spark/commit/07d42c5d61a3573579c163e59c5d49186e7573f4).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906617683
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47324/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906177581
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-938705690
**[Test build #144026 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144026/testReport)** for PR 33780 at commit [`9e14c21`](https://github.com/apache/spark/commit/9e14c21f8e2047ddbd4b8975d2fa5c71b097eff0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-938726072
**[Test build #144026 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144026/testReport)** for PR 33780 at commit [`9e14c21`](https://github.com/apache/spark/commit/9e14c21f8e2047ddbd4b8975d2fa5c71b097eff0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-940031824
merged to master, thanks everyone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906175183
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47292/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-900938364
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142598/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-909315224
> Like mentioned in the other issue, I think we should start discussion on spark dev..
The email has been sent, if I missed anything, please add it. Thanks a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-901048964
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47106/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-921845784
> Ok, I talked to a few other people and no one has strong opinions and I'm ok with the config as long as its very clearly documented that could get wrong failure.
Thanks a lot, so do you have any other suggestion in current code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906144355
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142791/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-901004781
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47104/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-901037253
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-932349969
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48298/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-932327578
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143786/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-938633022
**[Test build #144024 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144024/testReport)** for PR 33780 at commit [`9e14c21`](https://github.com/apache/spark/commit/9e14c21f8e2047ddbd4b8975d2fa5c71b097eff0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-930356644
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-930331548
ping @tgravescs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-938708674
**[Test build #144029 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144029/testReport)** for PR 33780 at commit [`9e14c21`](https://github.com/apache/spark/commit/9e14c21f8e2047ddbd4b8975d2fa5c71b097eff0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-913401237
ping @mridulm @Ngone51
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-901002806
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47098/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906411969
I'm still hesitant on this, one I don't like more configs, two it seems like it could still give you the wrong result. I understand the use case though. I want to go back through some things to investigate a bit more.
If we were to do this it would definitely need more explanation in the config settings.
Have you tried using the unmanagedAM? I don't have much experience with it but should resolve this because AM no longer remote.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-900982416
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142607/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-909258399
Like mentioned in the other issue, I think we should start discussion on spark dev..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-900973331
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142603/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-900930760
**[Test build #142598 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142598/testReport)** for PR 33780 at commit [`7cdffa9`](https://github.com/apache/spark/commit/7cdffa9d7420de424bce22b8f1504deedd5dddec).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-900973259
**[Test build #142603 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142603/testReport)** for PR 33780 at commit [`2d9578c`](https://github.com/apache/spark/commit/2d9578ca92de3b459c34833847de47e7664de61b).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-903735011
the main thing here is to make sure we don't report fail when we aren't sure. I need to remember all the cases. I assume in this case you are running client batch mode? ie not pyspark or spark-shell
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906183567
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47294/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-900978026
**[Test build #142607 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142607/testReport)** for PR 33780 at commit [`dd56ddd`](https://github.com/apache/spark/commit/dd56dddfcf5d3aeab4a8e742d05f2fce3b187d20).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906170290
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47293/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-909071924
**[Test build #142878 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142878/testReport)** for PR 33780 at commit [`d90816e`](https://github.com/apache/spark/commit/d90816e4798ad0ada5af2a7f5e60a405e6b8e98d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-909090852
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142878/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-900984519
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-909090852
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-930429215
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48232/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-939621134
ping @tgravescs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-909053152
**[Test build #142878 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142878/testReport)** for PR 33780 at commit [`d90816e`](https://github.com/apache/spark/commit/d90816e4798ad0ada5af2a7f5e60a405e6b8e98d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-909053152
**[Test build #142878 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142878/testReport)** for PR 33780 at commit [`d90816e`](https://github.com/apache/spark/commit/d90816e4798ad0ada5af2a7f5e60a405e6b8e98d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #33780:
URL: https://github.com/apache/spark/pull/33780#discussion_r724994930
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -784,6 +784,9 @@ private[spark] class ApplicationMaster(
*/
private class AMEndpoint(override val rpcEnv: RpcEnv, driver: RpcEndpointRef)
extends RpcEndpoint with Logging {
+ private var shutdown = false
Review comment:
> this was not done, please update.
Sorry for my mistake, updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-938661804
**[Test build #144024 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144024/testReport)** for PR 33780 at commit [`9e14c21`](https://github.com/apache/spark/commit/9e14c21f8e2047ddbd4b8975d2fa5c71b097eff0).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-932327578
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143786/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-932374077
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48298/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #33780:
URL: https://github.com/apache/spark/pull/33780#discussion_r719454162
##########
File path: docs/running-on-yarn.md
##########
@@ -441,6 +441,20 @@ To use a custom metrics.properties for the application master and executors, upd
</td>
<td>1.6.0</td>
</tr>
+<tr>
+ <td><code>spark.yarn.am.clientModeTreatDisconnectAsFailed</code></td>
+ <td>false</td>
+ <td>
+ In managed yarn-client mode, when am disconnect with driver, am will finish application with SUCCESS final status since in
Review comment:
how about something like this:
Treat yarn-client unclean disconnects as failures. In yarn-client mode, normally the application will always finish with a final status of SUCCESS because in some cases, it is not possible to know if the Application was terminated intentionally by the user or if there was a real error. This config changes that behavior such that if the Application Master disconnects from the driver uncleanly (ie without the proper shutdown handshake) the application will terminate with a final status of FAILED. This will allow the caller to decide if it was truly a failure. Note that if this config is set and the user just terminate the client application badly it may show a status of FAILED when it wasn't really FAILED.
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -784,6 +784,9 @@ private[spark] class ApplicationMaster(
*/
private class AMEndpoint(override val rpcEnv: RpcEnv, driver: RpcEndpointRef)
extends RpcEndpoint with Logging {
+ private var shutdown = false
Review comment:
make shutdown @volatile
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -843,8 +848,13 @@ private[spark] class ApplicationMaster(
// In cluster mode or unmanaged am case, do not rely on the disassociated event to exit
// This avoids potentially reporting incorrect exit codes if the driver fails
if (!(isClusterMode || sparkConf.get(YARN_UNMANAGED_AM))) {
- logInfo(s"Driver terminated or disconnected! Shutting down. $remoteAddress")
- finish(FinalApplicationStatus.SUCCEEDED, ApplicationMaster.EXIT_SUCCESS)
+ if (shutdown || !clientModeTreatDisconnectAsFailed) {
+ logInfo(s"Driver terminated or disconnected! Shutting down. $remoteAddress")
+ finish(FinalApplicationStatus.SUCCEEDED, ApplicationMaster.EXIT_SUCCESS)
+ } else {
+ logError(s"Application Master lose connection with driver! Shutting down. $remoteAddress")
Review comment:
replace lose with lost
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-909090852
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-938847931
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48505/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-938633022
**[Test build #144024 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144024/testReport)** for PR 33780 at commit [`9e14c21`](https://github.com/apache/spark/commit/9e14c21f8e2047ddbd4b8975d2fa5c71b097eff0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-900930760
**[Test build #142598 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142598/testReport)** for PR 33780 at commit [`7cdffa9`](https://github.com/apache/spark/commit/7cdffa9d7420de424bce22b8f1504deedd5dddec).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-900978026
**[Test build #142607 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142607/testReport)** for PR 33780 at commit [`dd56ddd`](https://github.com/apache/spark/commit/dd56dddfcf5d3aeab4a8e742d05f2fce3b187d20).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906121166
> > The problem is that not everyone calls sc.stop... in interactive environments you also have people killing it weirdly, there were a bunch of shut down cases in client mode that made determining the real success/failure an issue.
> > It was intentionally decided to just always return success in those cases. If this properly handles those cases I'm fine with it but I need to go try to remember what they all were and make sure they are handled. We should not report FAIL unless its truly a failure.
>
> Got your point..I miss the case such as killing spark-shell weirdly...then how about:
>
> * add a configuration such as spark.yarn.clientModeAMDisconnect as Failed, when true, make it as failed, when false let it behavior as before? then let user to choose? Two company's scheduler platform meet such case, so we can have a choose. Since we always non't need to rerun pysaprk/spark-shell/spark-sql?
> * For pyspark/spark-shell/spark-sql such interactive mode, make it behavior as before and for normal batch job, make it failed when disconnected?
I prefer the first way, since always scheduler platform will care about this thing, if scheduler platform call spark-sql/spark-shell/pyspark etc, it will stop the application when finish computation. Only the interactive command line initiated by the user will be killed at will
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-900930760
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906553900
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142820/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-909133464
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47381/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-909090852
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142878/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906182592
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47294/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-903740725
> the main thing here is to make sure we don't report fail when we aren't sure. I need to remember all the cases. I assume in this case you are running client batch mode? ie not pyspark or spark-shell
Since for yarn-client mode.
1. there are some other component integration with spark use this exit code to judge if the job running well, the exit code need to be clear.
2. if we call sc.stop() in program's end or trigger by ShutfownHook, in driver side, it's no matter if MonitorThread get reply and call `sc.stop()`
3. If it's a good network env, AM should receive the Shutdown message and finish with SUCCESS.
4. If's a bad network, AM can't connect with driver, it always be a failed application, but know, it return SUCCESS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-909086253
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47381/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-901048964
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906152049
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142793/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906533575
**[Test build #142820 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142820/testReport)** for PR 33780 at commit [`af00e87`](https://github.com/apache/spark/commit/af00e8747987cb6b950d60aabd713082491f4408).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-903751410
The problem is that not everyone calls sc.stop... in interactive environments you also have people killing it weirdly, there were a bunch of shut down cases in client mode that made determining the real success/failure an issue.
It was intentionally decided to just always return success in those cases. If this properly handles those cases I'm fine with it but I need to go try to remember what they all were and make sure they are handled. We should not report FAIL unless its truly a failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906426188
Just a note, this has been brought up before: https://issues.apache.org/jira/browse/SPARK-15283?jql=project%20%3D%20SPARK%20AND%20text%20~%20yarn-client
There is also a bit of history for original implementation here: https://issues.apache.org/jira/browse/SPARK-3627
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-921803433
Ok, I talked to a few other people and no one has strong opinions and I'm ok with the config as long as its very clearly documented that could get wrong failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-930331548
ping @tgravescs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu edited a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu edited a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906128069
ping @yaooqinn @maropu Can you take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-900982361
**[Test build #142607 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142607/testReport)** for PR 33780 at commit [`dd56ddd`](https://github.com/apache/spark/commit/dd56dddfcf5d3aeab4a8e742d05f2fce3b187d20).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` protected class YarnSchedulerEndpoint(override val rpcEnv: RpcEnv)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906155470
**[Test build #142794 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142794/testReport)** for PR 33780 at commit [`07d42c5`](https://github.com/apache/spark/commit/07d42c5d61a3573579c163e59c5d49186e7573f4).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-900984519
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-938284255
ping @tgravescs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-938708674
**[Test build #144029 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144029/testReport)** for PR 33780 at commit [`9e14c21`](https://github.com/apache/spark/commit/9e14c21f8e2047ddbd4b8975d2fa5c71b097eff0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-938661890
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144024/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-900965526
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47098/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-906533575
**[Test build #142820 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142820/testReport)** for PR 33780 at commit [`af00e87`](https://github.com/apache/spark/commit/af00e8747987cb6b950d60aabd713082491f4408).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33780: [SPARK-36540][YARN]YARN-CLIENT mode should check Shutdown message when AMEndpoint disconencted
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33780:
URL: https://github.com/apache/spark/pull/33780#issuecomment-901004448
**[Test build #142608 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142608/testReport)** for PR 33780 at commit [`dd56ddd`](https://github.com/apache/spark/commit/dd56dddfcf5d3aeab4a8e742d05f2fce3b187d20).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org