You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/04 13:31:27 UTC
[GitHub] [spark] Ngone51 opened a new pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Ngone51 opened a new pull request #29579:
URL: https://github.com/apache/spark/pull/29579
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
The motivation of this PR is to avoid caching the removed decommissioned executors in `TaskSchedulerImpl`. The cache is introduced in https://github.com/apache/spark/pull/29422. The cache will hold the `isHostDecommissioned` info for a while. So if the task `FetchFailure` event comes after the executor loss event, `DAGScheduler` can still get the `isHostDecommissioned` from the cache and unregister the host shuffle map status when the host is decommissioned too.
This PR tries to achieve the same goal without the cache. Instead of saving the `workerLost` in `ExecutorUpdated` / `ExecutorDecommissionInfo` / `ExecutorDecommissionState`, we could save the `hostOpt` directly. When the host is decommissioned or lost too, the `hostOpt` can be a specific host address. Otherwise, it's `None` to indicate that only the executor is decommissioned or lost.
Now that we have the host info, we can also unregister the host shuffle map status when `executorLost` is triggered for the decommissioned executor.
Besides, this PR also includes a few cleanups around the touched code.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
It helps to unregister the shuffle map status earlier for both decommission and normal executor lost cases.
It also saves memory in `TaskSchedulerImpl` and simplifies the code a little bit.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
This PR only refactor the code. The original behaviour should be covered by `DecommissionWorkerSuite`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683518988
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-687058923
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-688329488
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-688411963
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #29579:
URL: https://github.com/apache/spark/pull/29579#discussion_r482852349
##########
File path: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
##########
@@ -390,16 +390,18 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
case Some(executorInfo) =>
// This must be synchronized because variables mutated
// in this block are read when requesting executors
- val killed = CoarseGrainedSchedulerBackend.this.synchronized {
+ val lossReason = CoarseGrainedSchedulerBackend.this.synchronized {
addressToExecutorId -= executorInfo.executorAddress
executorDataMap -= executorId
executorsPendingLossReason -= executorId
- executorsPendingDecommission -= executorId
- executorsPendingToRemove.remove(executorId).getOrElse(false)
+ val decommissioned = executorsPendingDecommission.remove(executorId)
+ executorsPendingToRemove
+ .remove(executorId).filter(killedByDriver => killedByDriver).map(_ => ExecutorKilled)
Review comment:
Yeah..looks better. (I actually tried both ways before and was hesitant between them...but I tried the scala style at the end..)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-687249566
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683307623
**[Test build #128019 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128019/testReport)** for PR 29579 at commit [`d5bc756`](https://github.com/apache/spark/commit/d5bc7560f32d59861885c6f40ec2597d680e4612).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-687129553
**[Test build #128297 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128297/testReport)** for PR 29579 at commit [`58add67`](https://github.com/apache/spark/commit/58add670f43818e556b5fbe441274c68c3adefef).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-687130626
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683308444
**[Test build #128019 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128019/testReport)** for PR 29579 at commit [`d5bc756`](https://github.com/apache/spark/commit/d5bc7560f32d59861885c6f40ec2597d680e4612).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class ExecutorDecommissionInfo(message: String, hostOpt: Option[String] = None)`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683371427
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-688411020
**[Test build #128356 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128356/testReport)** for PR 29579 at commit [`d246840`](https://github.com/apache/spark/commit/d2468407f3d02e5a191ad373b5f2201a930cbb09).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29579:
URL: https://github.com/apache/spark/pull/29579#discussion_r484187912
##########
File path: core/src/main/scala/org/apache/spark/deploy/DeployMessage.scala
##########
@@ -188,7 +188,7 @@ private[deploy] object DeployMessages {
}
case class ExecutorUpdated(id: Int, state: ExecutorState, message: Option[String],
- exitStatus: Option[Int], workerLost: Boolean)
+ exitStatus: Option[Int], workerHost: Option[String])
Review comment:
We should add some comments to explain what `workerHost` indicates here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-686386785
**[Test build #128251 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128251/testReport)** for PR 29579 at commit [`0c0749e`](https://github.com/apache/spark/commit/0c0749eb683749f4635b57a15bf15d19eebe34c5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-687058433
**[Test build #128297 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128297/testReport)** for PR 29579 at commit [`58add67`](https://github.com/apache/spark/commit/58add670f43818e556b5fbe441274c68c3adefef).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #29579:
URL: https://github.com/apache/spark/pull/29579#discussion_r484233558
##########
File path: core/src/main/scala/org/apache/spark/deploy/DeployMessage.scala
##########
@@ -188,7 +188,7 @@ private[deploy] object DeployMessages {
}
case class ExecutorUpdated(id: Int, state: ExecutorState, message: Option[String],
- exitStatus: Option[Int], workerLost: Boolean)
+ exitStatus: Option[Int], workerHost: Option[String])
Review comment:
In case of the decommission, the `Worker` doesn't really lose at the time the Master sends `ExecutorUpdated` to the driver. So I think it might not be such accurate.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683534969
Thank you for the quick response @agrawaldevesh .
> However, with this PR, it seems you are removing the "clear shuffle on fetch failure" part. It seems that you will wait for the heartbeat failure to occur and the host be lost, even if the downstream has signaled fetch failure.
I think this PR doesn't change the semantics. We still clear shuffle status on fetch failure as you can see the only change for fetch failure in DAGScheduler is:
```java
- .exists(_.isHostDecommissioned)
+ .exists(_.hostOpt.isDefined)
```
It the fetch failure comes first before the executor lost, DAGScheduler will still ask TaskSchedulerImpl for the decommission state and unregister the shuffle status then. While if the executor lost comes first, fetch failure becomes a NoOp on shuffle status unregister.
I think the only difference is that, before this PR, if the executor lost event comes first, it can only unregister shuffle map status on that executor, even if we know the host is also decommissioned. But now we can unregister the host shuffle status because we pass in the host info directly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-686480213
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683371758
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128024/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #29579:
URL: https://github.com/apache/spark/pull/29579#discussion_r482852832
##########
File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
##########
@@ -1989,15 +1989,15 @@ private[spark] class DAGScheduler(
*/
private[scheduler] def handleExecutorLost(
execId: String,
- workerLost: Boolean): Unit = {
+ hostOpt: Option[String]): Unit = {
Review comment:
I changed it to `workerHost`, so I guess we can keep the comment?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] agrawaldevesh commented on a change in pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
agrawaldevesh commented on a change in pull request #29579:
URL: https://github.com/apache/spark/pull/29579#discussion_r479868285
##########
File path: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
##########
@@ -390,16 +390,18 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
case Some(executorInfo) =>
// This must be synchronized because variables mutated
// in this block are read when requesting executors
- val killed = CoarseGrainedSchedulerBackend.this.synchronized {
+ val lossReason = CoarseGrainedSchedulerBackend.this.synchronized {
addressToExecutorId -= executorInfo.executorAddress
executorDataMap -= executorId
executorsPendingLossReason -= executorId
- executorsPendingDecommission -= executorId
- executorsPendingToRemove.remove(executorId).getOrElse(false)
+ val decommissioned = executorsPendingDecommission.remove(executorId)
+ executorsPendingToRemove
+ .remove(executorId).filter(killedByDriver => killedByDriver).map(_ => ExecutorKilled)
Review comment:
This scala statement is quite a mouthful ? Would an if condition make it simpler ? How about:
```
if (executorsPendingToRemove.remove(executorId).isDefined) {
ExecutorKilled
} else if (decommissioned) {
ExecutorDecommision()
} else {
reason
}
```
Personally, I find this easier to read.
The part I find confusing is the `.filter(killedByDriver => killedByDriver)`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683371757
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683372334
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683518868
@agrawaldevesh @cloud-fan @holdenk Please take a look, thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] agrawaldevesh commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
agrawaldevesh commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683539864
Thanks for the explanations! I will get back to you in like 2-3 days after playing with it locally. (I am on PTO tomorrow).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-687147092
retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683379999
**[Test build #128025 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128025/testReport)** for PR 29579 at commit [`47da0d7`](https://github.com/apache/spark/commit/47da0d7d9f6c76fc452c06f04eb84e595cc32943).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-687249566
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683551640
**[Test build #128064 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128064/testReport)** for PR 29579 at commit [`a0bc4f6`](https://github.com/apache/spark/commit/a0bc4f6db19c9633dd51de3347a3a606b71a3d38).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683372334
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683308454
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128019/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-689284555
thanks all!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] agrawaldevesh commented on a change in pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
agrawaldevesh commented on a change in pull request #29579:
URL: https://github.com/apache/spark/pull/29579#discussion_r479917376
##########
File path: core/src/main/scala/org/apache/spark/deploy/master/Master.scala
##########
@@ -909,9 +909,9 @@ private[deploy] class Master(
exec.application.driver.send(ExecutorUpdated(
exec.id, ExecutorState.DECOMMISSIONED,
Some("worker decommissioned"), None,
- // workerLost is being set to true here to let the driver know that the host (aka. worker)
+ // worker host is being set here to let the driver know that the host (aka. worker)
Review comment:
nit: can you reword the comment to be more accurate now :-)
##########
File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
##########
@@ -1989,15 +1989,15 @@ private[spark] class DAGScheduler(
*/
private[scheduler] def handleExecutorLost(
execId: String,
- workerLost: Boolean): Unit = {
+ hostOpt: Option[String]): Unit = {
Review comment:
Can you change this method's comment also if you decide to go with hostOpt instead of workerLost (perhaps you ought to consider my consider my comment on making workerLost itself be an Optional[String]). The comment still refers to "standalone worker"
##########
File path: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
##########
@@ -489,17 +491,7 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
decomInfo: ExecutorDecommissionInfo): Boolean = {
logInfo(s"Asking executor $executorId to decommissioning.")
- try {
- scheduler.executorDecommission(executorId, decomInfo)
- if (driverEndpoint != null) {
- logInfo("Propagating executor decommission to driver.")
- driverEndpoint.send(DecommissionExecutor(executorId, decomInfo))
- }
- } catch {
- case e: Exception =>
- logError(s"Unexpected error during decommissioning ${e.toString}", e)
- return false
- }
+ scheduler.executorDecommission(executorId, decomInfo)
Review comment:
Agreed !
##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##########
@@ -922,28 +910,8 @@ private[spark] class TaskSchedulerImpl(
synchronized {
// Don't bother noting decommissioning for executors that we don't know about
if (executorIdToHost.contains(executorId)) {
- val oldDecomStateOpt = executorsPendingDecommission.get(executorId)
- val newDecomState = if (oldDecomStateOpt.isEmpty) {
- // This is the first time we are hearing of decommissioning this executor,
- // so create a brand new state.
- ExecutorDecommissionState(
- clock.getTimeMillis(),
- decommissionInfo.isHostDecommissioned)
- } else {
- val oldDecomState = oldDecomStateOpt.get
- if (!oldDecomState.isHostDecommissioned && decommissionInfo.isHostDecommissioned) {
- // Only the cluster manager is allowed to send decommission messages with
- // isHostDecommissioned set. So the new decommissionInfo is from the cluster
- // manager and is thus authoritative. Flip isHostDecommissioned to true but keep the old
- // decommission start time.
- ExecutorDecommissionState(
- oldDecomState.startTime,
- isHostDecommissioned = true)
- } else {
- oldDecomState
- }
- }
- executorsPendingDecommission(executorId) = newDecomState
+ executorsPendingDecommission(executorId) =
+ ExecutorDecommissionState(clock.getTimeMillis(), decommissionInfo.hostOpt)
Review comment:
Agreed ! This cleanup helps.
##########
File path: core/src/main/scala/org/apache/spark/scheduler/ExecutorLossReason.scala
##########
@@ -69,5 +69,8 @@ case class ExecutorProcessLost(
*
* This is used by the task scheduler to remove state associated with the executor, but
* not yet fail any tasks that were running in the executor before the executor is "fully" lost.
+ *
+ * @param hostOpt it will be set by [[TaskSchedulerImpl]] when the host is decommissioned too
*/
-private [spark] object ExecutorDecommission extends ExecutorLossReason("Executor decommission.")
+private [spark] case class ExecutorDecommission(var hostOpt: Option[String] = None)
Review comment:
I am not a fan of this change of making the hostOpt be a var instead of a val. I think you only need this for line 932 in TaskSchedulerImpl. I am sure you would be able to accommodate that use case in a different way.
The reason I don't like it is because other ExecutorLossReason's are "messages" (for example ExecutorProcessLost) and these messages tend to be immutable. I think it's a bit hacky to have ExecutorDecommission masquerading as a message but then make it be mutable.
Even ExecutorDecommission is a message that the TaskSchedulerImpl enqueues into the event loop of the DAGScheduler.
##########
File path: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
##########
@@ -489,17 +491,7 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
decomInfo: ExecutorDecommissionInfo): Boolean = {
logInfo(s"Asking executor $executorId to decommissioning.")
- try {
- scheduler.executorDecommission(executorId, decomInfo)
- if (driverEndpoint != null) {
- logInfo("Propagating executor decommission to driver.")
- driverEndpoint.send(DecommissionExecutor(executorId, decomInfo))
- }
- } catch {
- case e: Exception =>
- logError(s"Unexpected error during decommissioning ${e.toString}", e)
- return false
- }
+ scheduler.executorDecommission(executorId, decomInfo)
// Send decommission message to the executor (it could have originated on the executor
Review comment:
nit: While you are here, can you please also close the parenthesis here :-)
##########
File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
##########
@@ -2366,11 +2366,12 @@ private[scheduler] class DAGSchedulerEventProcessLoop(dagScheduler: DAGScheduler
dagScheduler.handleExecutorAdded(execId, host)
case ExecutorLost(execId, reason) =>
- val workerLost = reason match {
- case ExecutorProcessLost(_, true, _) => true
- case _ => false
+ val hostOpt = reason match {
+ case ExecutorProcessLost(_, host, _) => host
+ case ExecutorDecommission(host) => host
Review comment:
nit: should this be ExecutorDecommission(hostOpt) ... to keep parity with the original name.
##########
File path: core/src/main/scala/org/apache/spark/deploy/DeployMessage.scala
##########
@@ -188,7 +188,7 @@ private[deploy] object DeployMessages {
}
case class ExecutorUpdated(id: Int, state: ExecutorState, message: Option[String],
- exitStatus: Option[Int], workerLost: Boolean)
+ exitStatus: Option[Int], hostOpt: Option[String])
Review comment:
We can also leave the name as workerLost and just make it be an Optional[String] ? In the spirit of minimal code change ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-687130640
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128297/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-688411963
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29579:
URL: https://github.com/apache/spark/pull/29579#discussion_r484188484
##########
File path: core/src/main/scala/org/apache/spark/deploy/DeployMessage.scala
##########
@@ -188,7 +188,7 @@ private[deploy] object DeployMessages {
}
case class ExecutorUpdated(id: Int, state: ExecutorState, message: Option[String],
- exitStatus: Option[Int], workerLost: Boolean)
+ exitStatus: Option[Int], workerHost: Option[String])
Review comment:
and how about `hostOfLostWorker`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683307623
**[Test build #128019 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128019/testReport)** for PR 29579 at commit [`d5bc756`](https://github.com/apache/spark/commit/d5bc7560f32d59861885c6f40ec2597d680e4612).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683371358
**[Test build #128024 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128024/testReport)** for PR 29579 at commit [`404f92b`](https://github.com/apache/spark/commit/404f92b85cc6f9e5dadaa8301a4b9dd6024a143c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #29579:
URL: https://github.com/apache/spark/pull/29579
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-686480213
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 edited a comment on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
Ngone51 edited a comment on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683534969
Thank you for the quick response @agrawaldevesh .
> However, with this PR, it seems you are removing the "clear shuffle on fetch failure" part. It seems that you will wait for the heartbeat failure to occur and the host be lost, even if the downstream has signaled fetch failure.
I think this PR doesn't change the semantics. We still clear shuffle status on fetch failure as you can see the only change for fetch failure in DAGScheduler is:
```diff
- .exists(_.isHostDecommissioned)
+ .exists(_.hostOpt.isDefined)
```
If the fetch failure comes first before the executor lost, DAGScheduler will still ask TaskSchedulerImpl for the decommission state and unregister the shuffle status then. While if the executor lost comes first, fetch failure becomes a NoOp on shuffle status unregister.
I think the only difference is that, before this PR, if the executor lost event comes first, it can only unregister shuffle map status on that executor, even if we know the host is also decommissioned. But now we can unregister the host shuffle status because we pass in the host info directly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-688328510
**[Test build #128356 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128356/testReport)** for PR 29579 at commit [`d246840`](https://github.com/apache/spark/commit/d2468407f3d02e5a191ad373b5f2201a930cbb09).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683371427
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683307753
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683518653
**[Test build #128064 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128064/testReport)** for PR 29579 at commit [`a0bc4f6`](https://github.com/apache/spark/commit/a0bc4f6db19c9633dd51de3347a3a606b71a3d38).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-686387532
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683371358
**[Test build #128024 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128024/testReport)** for PR 29579 at commit [`404f92b`](https://github.com/apache/spark/commit/404f92b85cc6f9e5dadaa8301a4b9dd6024a143c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-688328510
**[Test build #128356 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128356/testReport)** for PR 29579 at commit [`d246840`](https://github.com/apache/spark/commit/d2468407f3d02e5a191ad373b5f2201a930cbb09).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-687984920
@cloud-fan @holdenk Could you take a look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683307753
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683552127
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128064/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-687248374
**[Test build #128305 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128305/testReport)** for PR 29579 at commit [`58add67`](https://github.com/apache/spark/commit/58add670f43818e556b5fbe441274c68c3adefef).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-686478252
**[Test build #128251 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128251/testReport)** for PR 29579 at commit [`0c0749e`](https://github.com/apache/spark/commit/0c0749eb683749f4635b57a15bf15d19eebe34c5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-686386785
**[Test build #128251 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128251/testReport)** for PR 29579 at commit [`0c0749e`](https://github.com/apache/spark/commit/0c0749eb683749f4635b57a15bf15d19eebe34c5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #29579:
URL: https://github.com/apache/spark/pull/29579#discussion_r482851318
##########
File path: core/src/main/scala/org/apache/spark/deploy/DeployMessage.scala
##########
@@ -188,7 +188,7 @@ private[deploy] object DeployMessages {
}
case class ExecutorUpdated(id: Int, state: ExecutorState, message: Option[String],
- exitStatus: Option[Int], workerLost: Boolean)
+ exitStatus: Option[Int], hostOpt: Option[String])
Review comment:
After a second thought, I changed it to `workerHost`. We need the keyword `worker` because it's specific to Standalone Worker. And `Host` gives the direct meaning of the value. And `workerLost` sounds more appropriate for the `Boolean` type. WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] agrawaldevesh commented on a change in pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
agrawaldevesh commented on a change in pull request #29579:
URL: https://github.com/apache/spark/pull/29579#discussion_r479868581
##########
File path: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
##########
@@ -489,17 +491,7 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
decomInfo: ExecutorDecommissionInfo): Boolean = {
logInfo(s"Asking executor $executorId to decommissioning.")
- try {
- scheduler.executorDecommission(executorId, decomInfo)
- if (driverEndpoint != null) {
- logInfo("Propagating executor decommission to driver.")
- driverEndpoint.send(DecommissionExecutor(executorId, decomInfo))
- }
- } catch {
- case e: Exception =>
- logError(s"Unexpected error during decommissioning ${e.toString}", e)
- return false
- }
+ scheduler.executorDecommission(executorId, decomInfo)
Review comment:
Just trying to follow why the driver end point message was dropped ?
##########
File path: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
##########
@@ -390,16 +390,18 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
case Some(executorInfo) =>
// This must be synchronized because variables mutated
// in this block are read when requesting executors
- val killed = CoarseGrainedSchedulerBackend.this.synchronized {
+ val lossReason = CoarseGrainedSchedulerBackend.this.synchronized {
addressToExecutorId -= executorInfo.executorAddress
executorDataMap -= executorId
executorsPendingLossReason -= executorId
- executorsPendingDecommission -= executorId
- executorsPendingToRemove.remove(executorId).getOrElse(false)
+ val decommissioned = executorsPendingDecommission.remove(executorId)
+ executorsPendingToRemove
+ .remove(executorId).filter(killedByDriver => killedByDriver).map(_ => ExecutorKilled)
Review comment:
This scala statement is quite a mouthful ? Would an if condition make it simpler ?
##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##########
@@ -922,28 +910,8 @@ private[spark] class TaskSchedulerImpl(
synchronized {
// Don't bother noting decommissioning for executors that we don't know about
if (executorIdToHost.contains(executorId)) {
- val oldDecomStateOpt = executorsPendingDecommission.get(executorId)
- val newDecomState = if (oldDecomStateOpt.isEmpty) {
- // This is the first time we are hearing of decommissioning this executor,
- // so create a brand new state.
- ExecutorDecommissionState(
- clock.getTimeMillis(),
- decommissionInfo.isHostDecommissioned)
- } else {
- val oldDecomState = oldDecomStateOpt.get
- if (!oldDecomState.isHostDecommissioned && decommissionInfo.isHostDecommissioned) {
- // Only the cluster manager is allowed to send decommission messages with
- // isHostDecommissioned set. So the new decommissionInfo is from the cluster
- // manager and is thus authoritative. Flip isHostDecommissioned to true but keep the old
- // decommission start time.
- ExecutorDecommissionState(
- oldDecomState.startTime,
- isHostDecommissioned = true)
- } else {
- oldDecomState
- }
- }
- executorsPendingDecommission(executorId) = newDecomState
+ executorsPendingDecommission(executorId) =
+ ExecutorDecommissionState(clock.getTimeMillis(), decommissionInfo.hostOpt)
Review comment:
Just to make sure I am understanding: are you removing the multiple message handling ? Yeah we do not have good coverage on that but why remove it ?
##########
File path: core/src/main/scala/org/apache/spark/deploy/DeployMessage.scala
##########
@@ -188,7 +188,7 @@ private[deploy] object DeployMessages {
}
case class ExecutorUpdated(id: Int, state: ExecutorState, message: Option[String],
- exitStatus: Option[Int], workerLost: Boolean)
+ exitStatus: Option[Int], hostOpt: Option[String])
Review comment:
I think we need a better name than hostOpt ? How about just "Hostname" ? The type already conveys that this is an optional.
##########
File path: core/src/main/scala/org/apache/spark/deploy/DeployMessage.scala
##########
@@ -188,7 +188,7 @@ private[deploy] object DeployMessages {
}
case class ExecutorUpdated(id: Int, state: ExecutorState, message: Option[String],
- exitStatus: Option[Int], workerLost: Boolean)
+ exitStatus: Option[Int], hostOpt: Option[String])
Review comment:
How about hostLost ?
##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##########
@@ -952,28 +920,19 @@ private[spark] class TaskSchedulerImpl(
override def getExecutorDecommissionState(executorId: String)
: Option[ExecutorDecommissionState] = synchronized {
- executorsPendingDecommission
- .get(executorId)
- .orElse(Option(decommissionedExecutorsRemoved.get(executorId)))
+ executorsPendingDecommission.get(executorId)
}
- override def executorLost(executorId: String, givenReason: ExecutorLossReason): Unit = {
+ override def executorLost(executorId: String, reason: ExecutorLossReason): Unit = {
var failedExecutor: Option[String] = None
- val reason = givenReason match {
- // Handle executor process loss due to decommissioning
- case ExecutorProcessLost(message, origWorkerLost, origCausedByApp) =>
- val executorDecommissionState = getExecutorDecommissionState(executorId)
- ExecutorProcessLost(
- message,
- // Also mark the worker lost if we know that the host was decommissioned
- origWorkerLost || executorDecommissionState.exists(_.isHostDecommissioned),
- // Executor loss is certainly not caused by app if we knew that this executor is being
- // decommissioned
- causedByApp = executorDecommissionState.isEmpty && origCausedByApp)
- case e => e
- }
synchronized {
+ reason match {
+ case e @ ExecutorDecommission(_) =>
+ e.hostOpt = getExecutorDecommissionState(executorId).map(_.hostOpt).get
Review comment:
Is hostOpt a var that it can be thus changed ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] agrawaldevesh commented on a change in pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
agrawaldevesh commented on a change in pull request #29579:
URL: https://github.com/apache/spark/pull/29579#discussion_r483110363
##########
File path: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
##########
@@ -92,8 +92,9 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
// Executors that have been lost, but for which we don't yet know the real exit reason.
private val executorsPendingLossReason = new HashSet[String]
- // Executors which are being decommissioned
- protected val executorsPendingDecommission = new HashSet[String]
+ // Executors which are being decommissioned. Maps from executorId to
+ // workerHost(it's defined when the worker is also decommissioned)
Review comment:
super nit: space after workerHost.
I think workerHost is already an Option and thus already matches the value type of the executorsPendingDecommission map. Thus, we can perhaps drop the parenthesis clause entirely ?
##########
File path: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
##########
@@ -394,10 +395,15 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
addressToExecutorId -= executorInfo.executorAddress
executorDataMap -= executorId
executorsPendingLossReason -= executorId
+ val killedByDriver = executorsPendingToRemove.remove(executorId).getOrElse(false)
val decommissioned = executorsPendingDecommission.remove(executorId)
- executorsPendingToRemove
- .remove(executorId).filter(killedByDriver => killedByDriver).map(_ => ExecutorKilled)
- .getOrElse(if (decommissioned) ExecutorDecommission() else reason)
+ if (killedByDriver) {
+ ExecutorKilled
+ } else if (decommissioned.isDefined) {
+ ExecutorDecommission(decommissioned.get)
+ } else {
+ reason
+ }
Review comment:
:-) I can read !.
##########
File path: core/src/main/scala/org/apache/spark/scheduler/ExecutorDecommissionInfo.scala
##########
@@ -20,12 +20,12 @@ package org.apache.spark.scheduler
/**
* Message providing more detail when an executor is being decommissioned.
* @param message Human readable reason for why the decommissioning is happening.
- * @param isHostDecommissioned Whether the host (aka the `node` or `worker` in other places) is
- * being decommissioned too. Used to infer if the shuffle data might
- * be lost even if the external shuffle service is enabled.
+ * @param workerHost When workerHost is defined. It means the host (aka the `node` or `worker`
Review comment:
nit: When workerHost is defined, it means ...
(comma instead of full stop)
##########
File path: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
##########
@@ -394,10 +395,15 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
addressToExecutorId -= executorInfo.executorAddress
executorDataMap -= executorId
executorsPendingLossReason -= executorId
+ val killedByDriver = executorsPendingToRemove.remove(executorId).getOrElse(false)
val decommissioned = executorsPendingDecommission.remove(executorId)
Review comment:
Rename decommissioned to workerHostOpt and perhaps give it an explicit type: Option[Option[String]]. Its no longer a simple boolean.
##########
File path: core/src/main/scala/org/apache/spark/deploy/client/StandaloneAppClient.scala
##########
@@ -175,15 +175,15 @@ private[spark] class StandaloneAppClient(
cores))
listener.executorAdded(fullId, workerId, hostPort, cores, memory)
- case ExecutorUpdated(id, state, message, exitStatus, workerLost) =>
+ case ExecutorUpdated(id, state, message, exitStatus, workerHost) =>
Review comment:
Personally, I would still be okay with workerLost being an Option[String] instead of a Boolean. Obviously, had it been called "workerIsLost" then we would have to rename it. But I am also fine with the new name workerHost as well. I don't particularly think that the name workerLost must connote a boolean.
This ExecutorUpdated message is a case in point where the "lost" part is meaningful because it refers to the "worker that is lost" as opposed to some random worker-host.
But no strong feelings on this and I am happy with the choice workerHost.
##########
File path: core/src/main/scala/org/apache/spark/scheduler/ExecutorLossReason.scala
##########
@@ -70,7 +71,7 @@ case class ExecutorProcessLost(
* This is used by the task scheduler to remove state associated with the executor, but
* not yet fail any tasks that were running in the executor before the executor is "fully" lost.
*
- * @param hostOpt it will be set by [[TaskSchedulerImpl]] when the host is decommissioned too
+ * @param workerHost it's defined when the worker is decommissioned too
Review comment:
nit: "it's" -> "it is"
Also, should we explicitly bring out the word 'host' here ? "It is defined when the worker _host_ is decommissioned too"
##########
File path: core/src/main/scala/org/apache/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
##########
@@ -165,10 +165,13 @@ private[spark] class StandaloneSchedulerBackend(
}
override def executorRemoved(
- fullId: String, message: String, exitStatus: Option[Int], hostOpt: Option[String]): Unit = {
+ fullId: String,
+ message: String,
+ exitStatus: Option[Int],
+ workerHost: Option[String]): Unit = {
Review comment:
nit: Extra space before workerHost
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-688614265
thanks, merging to master!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683371752
**[Test build #128024 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128024/testReport)** for PR 29579 at commit [`404f92b`](https://github.com/apache/spark/commit/404f92b85cc6f9e5dadaa8301a4b9dd6024a143c).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-687058433
**[Test build #128297 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128297/testReport)** for PR 29579 at commit [`58add67`](https://github.com/apache/spark/commit/58add670f43818e556b5fbe441274c68c3adefef).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-688329488
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #29579:
URL: https://github.com/apache/spark/pull/29579#discussion_r482854249
##########
File path: core/src/main/scala/org/apache/spark/scheduler/ExecutorLossReason.scala
##########
@@ -69,5 +69,8 @@ case class ExecutorProcessLost(
*
* This is used by the task scheduler to remove state associated with the executor, but
* not yet fail any tasks that were running in the executor before the executor is "fully" lost.
+ *
+ * @param hostOpt it will be set by [[TaskSchedulerImpl]] when the host is decommissioned too
*/
-private [spark] object ExecutorDecommission extends ExecutorLossReason("Executor decommission.")
+private [spark] case class ExecutorDecommission(var hostOpt: Option[String] = None)
Review comment:
TBH I don't like the way myself too. I tried another way to get rid of the problem here but requires storing the redundant `workHost` info at `CoarseGrainedSchedulerBackend`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683372701
**[Test build #128025 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128025/testReport)** for PR 29579 at commit [`47da0d7`](https://github.com/apache/spark/commit/47da0d7d9f6c76fc452c06f04eb84e595cc32943).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683380078
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128025/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683372701
**[Test build #128025 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128025/testReport)** for PR 29579 at commit [`47da0d7`](https://github.com/apache/spark/commit/47da0d7d9f6c76fc452c06f04eb84e595cc32943).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-687151927
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-686387532
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-687130626
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #29579:
URL: https://github.com/apache/spark/pull/29579#discussion_r479876219
##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##########
@@ -922,28 +910,8 @@ private[spark] class TaskSchedulerImpl(
synchronized {
// Don't bother noting decommissioning for executors that we don't know about
if (executorIdToHost.contains(executorId)) {
- val oldDecomStateOpt = executorsPendingDecommission.get(executorId)
- val newDecomState = if (oldDecomStateOpt.isEmpty) {
- // This is the first time we are hearing of decommissioning this executor,
- // so create a brand new state.
- ExecutorDecommissionState(
- clock.getTimeMillis(),
- decommissionInfo.isHostDecommissioned)
- } else {
- val oldDecomState = oldDecomStateOpt.get
- if (!oldDecomState.isHostDecommissioned && decommissionInfo.isHostDecommissioned) {
- // Only the cluster manager is allowed to send decommission messages with
- // isHostDecommissioned set. So the new decommissionInfo is from the cluster
- // manager and is thus authoritative. Flip isHostDecommissioned to true but keep the old
- // decommission start time.
- ExecutorDecommissionState(
- oldDecomState.startTime,
- isHostDecommissioned = true)
- } else {
- oldDecomState
- }
- }
- executorsPendingDecommission(executorId) = newDecomState
+ executorsPendingDecommission(executorId) =
+ ExecutorDecommissionState(clock.getTimeMillis(), decommissionInfo.hostOpt)
Review comment:
The multiple decommission messages for the same executor shouldn't happen as it has been guaranteed by `CoarseGrainedSchedulerBackend.decommissionExecutors`:
https://github.com/apache/spark/blob/a1e459ed9f6777fb8d5a2d09fda666402f9230b9/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L465-L475
we won't decommission an executor which is already decommissioned(in-active).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683552120
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683380076
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683552120
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683308453
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683308453
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683518653
**[Test build #128064 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128064/testReport)** for PR 29579 at commit [`a0bc4f6`](https://github.com/apache/spark/commit/a0bc4f6db19c9633dd51de3347a3a606b71a3d38).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #29579:
URL: https://github.com/apache/spark/pull/29579#discussion_r482852617
##########
File path: core/src/main/scala/org/apache/spark/deploy/master/Master.scala
##########
@@ -909,9 +909,9 @@ private[deploy] class Master(
exec.application.driver.send(ExecutorUpdated(
exec.id, ExecutorState.DECOMMISSIONED,
Some("worker decommissioned"), None,
- // workerLost is being set to true here to let the driver know that the host (aka. worker)
+ // worker host is being set here to let the driver know that the host (aka. worker)
Review comment:
Updated a little bit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683371757
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] agrawaldevesh commented on a change in pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
agrawaldevesh commented on a change in pull request #29579:
URL: https://github.com/apache/spark/pull/29579#discussion_r479867792
##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##########
@@ -952,28 +920,19 @@ private[spark] class TaskSchedulerImpl(
override def getExecutorDecommissionState(executorId: String)
: Option[ExecutorDecommissionState] = synchronized {
- executorsPendingDecommission
- .get(executorId)
- .orElse(Option(decommissionedExecutorsRemoved.get(executorId)))
+ executorsPendingDecommission.get(executorId)
}
- override def executorLost(executorId: String, givenReason: ExecutorLossReason): Unit = {
+ override def executorLost(executorId: String, reason: ExecutorLossReason): Unit = {
var failedExecutor: Option[String] = None
- val reason = givenReason match {
- // Handle executor process loss due to decommissioning
- case ExecutorProcessLost(message, origWorkerLost, origCausedByApp) =>
- val executorDecommissionState = getExecutorDecommissionState(executorId)
- ExecutorProcessLost(
- message,
- // Also mark the worker lost if we know that the host was decommissioned
- origWorkerLost || executorDecommissionState.exists(_.isHostDecommissioned),
- // Executor loss is certainly not caused by app if we knew that this executor is being
- // decommissioned
- causedByApp = executorDecommissionState.isEmpty && origCausedByApp)
- case e => e
- }
synchronized {
+ reason match {
+ case e @ ExecutorDecommission(_) =>
+ e.hostOpt = getExecutorDecommissionState(executorId).map(_.hostOpt).get
Review comment:
Is hostOpt a var that it can be thus changed ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #29579:
URL: https://github.com/apache/spark/pull/29579#discussion_r479877389
##########
File path: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
##########
@@ -489,17 +491,7 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
decomInfo: ExecutorDecommissionInfo): Boolean = {
logInfo(s"Asking executor $executorId to decommissioning.")
- try {
- scheduler.executorDecommission(executorId, decomInfo)
- if (driverEndpoint != null) {
- logInfo("Propagating executor decommission to driver.")
- driverEndpoint.send(DecommissionExecutor(executorId, decomInfo))
- }
- } catch {
- case e: Exception =>
- logError(s"Unexpected error during decommissioning ${e.toString}", e)
- return false
- }
+ scheduler.executorDecommission(executorId, decomInfo)
Review comment:
At this point, the executor has been added to `executorsPendingDecommission`, which is considered to be decommissioned already. The message `DecommissionExecutor` calls `CoarseGrainedSchedulerBackend.decommissionExecutors` indeed. However, it's a NoOp for an executor that is already decommissioned.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683518988
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-687151927
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #29579:
URL: https://github.com/apache/spark/pull/29579#discussion_r483526519
##########
File path: core/src/main/scala/org/apache/spark/scheduler/ExecutorLossReason.scala
##########
@@ -70,7 +71,7 @@ case class ExecutorProcessLost(
* This is used by the task scheduler to remove state associated with the executor, but
* not yet fail any tasks that were running in the executor before the executor is "fully" lost.
*
- * @param hostOpt it will be set by [[TaskSchedulerImpl]] when the host is decommissioned too
+ * @param workerHost it's defined when the worker is decommissioned too
Review comment:
I think "worker" should be enough.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-687151154
**[Test build #128305 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128305/testReport)** for PR 29579 at commit [`58add67`](https://github.com/apache/spark/commit/58add670f43818e556b5fbe441274c68c3adefef).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 edited a comment on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
Ngone51 edited a comment on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683534969
Thank you for the quick response @agrawaldevesh .
> However, with this PR, it seems you are removing the "clear shuffle on fetch failure" part. It seems that you will wait for the heartbeat failure to occur and the host be lost, even if the downstream has signaled fetch failure.
I think this PR doesn't change the semantics. We still clear shuffle status on fetch failure as you can see the only change for fetch failure in DAGScheduler is:
```java
- .exists(_.isHostDecommissioned)
+ .exists(_.hostOpt.isDefined)
```
If the fetch failure comes first before the executor lost, DAGScheduler will still ask TaskSchedulerImpl for the decommission state and unregister the shuffle status then. While if the executor lost comes first, fetch failure becomes a NoOp on shuffle status unregister.
I think the only difference is that, before this PR, if the executor lost event comes first, it can only unregister shuffle map status on that executor, even if we know the host is also decommissioned. But now we can unregister the host shuffle status because we pass in the host info directly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-683380076
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-686383921
@cloud-fan @holdenk Could you also take a look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 closed pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
Ngone51 closed pull request #29579:
URL: https://github.com/apache/spark/pull/29579
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-687151154
**[Test build #128305 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128305/testReport)** for PR 29579 at commit [`58add67`](https://github.com/apache/spark/commit/58add670f43818e556b5fbe441274c68c3adefef).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #29579:
URL: https://github.com/apache/spark/pull/29579#discussion_r483525890
##########
File path: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
##########
@@ -394,10 +395,15 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
addressToExecutorId -= executorInfo.executorAddress
executorDataMap -= executorId
executorsPendingLossReason -= executorId
+ val killedByDriver = executorsPendingToRemove.remove(executorId).getOrElse(false)
val decommissioned = executorsPendingDecommission.remove(executorId)
Review comment:
Rename to `workerHostOpt` make sense to me. But I don't have a strong feeling to add the explicit type. It also breaks one line length limitation. I'd like to keep it in one line when it's not necessary to break it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29579: [SPARK-32736][CORE] Avoid caching the removed decommissioned executors in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29579:
URL: https://github.com/apache/spark/pull/29579#issuecomment-687058923
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org