You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/08/18 16:54:15 UTC
[GitHub] [spark] Ngone51 opened a new pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Ngone51 opened a new pull request #29468:
URL: https://github.com/apache/spark/pull/29468
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Add decommissioning status checking for a host or executor while checking it's active or not. And a decommissioned host or executor should be considered as inactive.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
The active status checking for host and executor in `TaskSchedulerImpl` is used to calculate available preferred locations in `TaskSetManager`. However, the decommissioned host or executor will no longer be used to assign any tasks. Thus, `TaskSchedulerImpl` should also prevent the decommissioned host or executor to be used by `TaskSetManager`.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Added a unit test. (I'll try to add a unit test for `TaskSetManager` later)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-676186374
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-675914996
**[Test build #127636 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127636/testReport)** for PR 29468 at commit [`aa2d5ba`](https://github.com/apache/spark/commit/aa2d5baf5983d80be491d3264b5ef80ad4c6b51b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-676191847
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-675917559
> Now all we need is a real test to validate that the PR is actually fixing the delay scheduling issue you pointed out.
@agrawaldevesh I added a unit test for locality level computation in `TaskSetManagerSuite`. I'm not sure what do you mean by a "real test". But if it means an end to end test which validates this fix can avoid the unnecessary delay, then I think it can be very hard.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-676190065
**[Test build #127648 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127648/testReport)** for PR 29468 at commit [`aa2d5ba`](https://github.com/apache/spark/commit/aa2d5baf5983d80be491d3264b5ef80ad4c6b51b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] agrawaldevesh commented on a change in pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
agrawaldevesh commented on a change in pull request #29468:
URL: https://github.com/apache/spark/pull/29468#discussion_r472762189
##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##########
@@ -1062,25 +1062,36 @@ private[spark] class TaskSchedulerImpl(
}
def getExecutorsAliveOnHost(host: String): Option[Set[String]] = synchronized {
- hostToExecutors.get(host).map(_.toSet)
+ hostToExecutors.get(host).map(_.filterNot(isExecutorDecommissioned)).map(_.toSet)
Review comment:
I am just a bit wary of regressions and I am not too sure about the unit test coverage here. So generally like the code changes to be minimal. But I am okay with this change if you think it is safe.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-675595569
@cloud-fan @agrawaldevesh @holdenk Please take a look, thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-677392955
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] agrawaldevesh commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
agrawaldevesh commented on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-675883231
Thanks for updating the PR description. Now all we need is a real test to validate that the PR is actually fixing the delay scheduling issue you pointed out. The current test is not really testing that but rather just testing the "definition of aliveness" -- so it is good to retain as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-677506332
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-675914996
**[Test build #127636 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127636/testReport)** for PR 29468 at commit [`aa2d5ba`](https://github.com/apache/spark/commit/aa2d5baf5983d80be491d3264b5ef80ad4c6b51b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-675597350
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-675830001
Have synced with @agrawaldevesh offline, I've updated the PR description according to his questions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #29468:
URL: https://github.com/apache/spark/pull/29468
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-675596852
**[Test build #127592 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127592/testReport)** for PR 29468 at commit [`a4eaf5f`](https://github.com/apache/spark/commit/a4eaf5f5cd04ca3efc7de08b77224e35ca1fd5fe).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-675915430
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-675678163
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] agrawaldevesh commented on a change in pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
agrawaldevesh commented on a change in pull request #29468:
URL: https://github.com/apache/spark/pull/29468#discussion_r472760587
##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##########
@@ -1062,25 +1062,36 @@ private[spark] class TaskSchedulerImpl(
}
def getExecutorsAliveOnHost(host: String): Option[Set[String]] = synchronized {
- hostToExecutors.get(host).map(_.toSet)
+ hostToExecutors.get(host).map(_.filterNot(isExecutorDecommissioned)).map(_.toSet)
}
def hasExecutorsAliveOnHost(host: String): Boolean = synchronized {
- hostToExecutors.contains(host)
+ hostToExecutors.get(host)
+ .exists(executors => executors.exists(e => !isExecutorDecommissioned(e)))
}
def hasHostAliveOnRack(rack: String): Boolean = synchronized {
- hostsByRack.contains(rack)
+ hostsByRack.get(rack)
Review comment:
Got it. Thanks for considering it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-677271793
**[Test build #127677 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127677/testReport)** for PR 29468 at commit [`65f6e05`](https://github.com/apache/spark/commit/65f6e0525b2d1472c1e6956587253e5ae6263676).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-677506332
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-677382791
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] agrawaldevesh commented on a change in pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
agrawaldevesh commented on a change in pull request #29468:
URL: https://github.com/apache/spark/pull/29468#discussion_r472366419
##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##########
@@ -1062,25 +1062,36 @@ private[spark] class TaskSchedulerImpl(
}
def getExecutorsAliveOnHost(host: String): Option[Set[String]] = synchronized {
- hostToExecutors.get(host).map(_.toSet)
+ hostToExecutors.get(host).map(_.filterNot(isExecutorDecommissioned)).map(_.toSet)
}
def hasExecutorsAliveOnHost(host: String): Boolean = synchronized {
- hostToExecutors.contains(host)
+ hostToExecutors.get(host)
+ .exists(executors => executors.exists(e => !isExecutorDecommissioned(e)))
}
def hasHostAliveOnRack(rack: String): Boolean = synchronized {
- hostsByRack.contains(rack)
+ hostsByRack.get(rack)
Review comment:
I haven't looked at this code too deeply, but does it make sense it eagerly remove the decommissioned executor from hostToExecutors and recomputeLocality and thus rebuild hostsByRack etc ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-676433269
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-675597350
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29468:
URL: https://github.com/apache/spark/pull/29468#discussion_r472786963
##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##########
@@ -1062,25 +1062,36 @@ private[spark] class TaskSchedulerImpl(
}
def getExecutorsAliveOnHost(host: String): Option[Set[String]] = synchronized {
- hostToExecutors.get(host).map(_.toSet)
+ hostToExecutors.get(host).map(_.filterNot(isExecutorDecommissioned)).map(_.toSet)
Review comment:
It's super weird if `getExecutorsAliveOnHost` and `hasExecutorsAliveOnHost` are not consistent.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #29468:
URL: https://github.com/apache/spark/pull/29468#discussion_r473641668
##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##########
@@ -1062,25 +1062,38 @@ private[spark] class TaskSchedulerImpl(
}
def getExecutorsAliveOnHost(host: String): Option[Set[String]] = synchronized {
- hostToExecutors.get(host).map(_.toSet)
+ hostToExecutors.get(host).map(_.filterNot(isExecutorDecommissioned)).map(_.toSet)
}
def hasExecutorsAliveOnHost(host: String): Boolean = synchronized {
- hostToExecutors.contains(host)
+ hostToExecutors.get(host)
+ .exists(executors => executors.exists(e => !isExecutorDecommissioned(e)))
}
def hasHostAliveOnRack(rack: String): Boolean = synchronized {
- hostsByRack.contains(rack)
+ hostsByRack.get(rack)
+ .exists(hosts => hosts.exists(h => !isHostDecommissioned(h)))
}
def isExecutorAlive(execId: String): Boolean = synchronized {
- executorIdToRunningTaskIds.contains(execId)
+ executorIdToRunningTaskIds.contains(execId) && !isExecutorDecommissioned(execId)
}
def isExecutorBusy(execId: String): Boolean = synchronized {
executorIdToRunningTaskIds.get(execId).exists(_.nonEmpty)
}
+ // exposed for test
+ protected def isExecutorDecommissioned(execId: String): Boolean =
+ getExecutorDecommissionInfo(execId).nonEmpty || isHostDecommissioned(executorIdToHost(execId))
Review comment:
make sense to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-676190065
**[Test build #127648 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127648/testReport)** for PR 29468 at commit [`aa2d5ba`](https://github.com/apache/spark/commit/aa2d5baf5983d80be491d3264b5ef80ad4c6b51b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-676433294
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127648/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] agrawaldevesh commented on a change in pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
agrawaldevesh commented on a change in pull request #29468:
URL: https://github.com/apache/spark/pull/29468#discussion_r473172376
##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##########
@@ -1062,25 +1062,38 @@ private[spark] class TaskSchedulerImpl(
}
def getExecutorsAliveOnHost(host: String): Option[Set[String]] = synchronized {
- hostToExecutors.get(host).map(_.toSet)
+ hostToExecutors.get(host).map(_.filterNot(isExecutorDecommissioned)).map(_.toSet)
}
def hasExecutorsAliveOnHost(host: String): Boolean = synchronized {
- hostToExecutors.contains(host)
+ hostToExecutors.get(host)
+ .exists(executors => executors.exists(e => !isExecutorDecommissioned(e)))
}
def hasHostAliveOnRack(rack: String): Boolean = synchronized {
- hostsByRack.contains(rack)
+ hostsByRack.get(rack)
+ .exists(hosts => hosts.exists(h => !isHostDecommissioned(h)))
}
def isExecutorAlive(execId: String): Boolean = synchronized {
- executorIdToRunningTaskIds.contains(execId)
+ executorIdToRunningTaskIds.contains(execId) && !isExecutorDecommissioned(execId)
}
def isExecutorBusy(execId: String): Boolean = synchronized {
executorIdToRunningTaskIds.get(execId).exists(_.nonEmpty)
}
+ // exposed for test
Review comment:
nit: Should these two methods be declared final : We don't really want to override them and instead want to override the helper methods it calls.
##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##########
@@ -1062,25 +1062,38 @@ private[spark] class TaskSchedulerImpl(
}
def getExecutorsAliveOnHost(host: String): Option[Set[String]] = synchronized {
- hostToExecutors.get(host).map(_.toSet)
+ hostToExecutors.get(host).map(_.filterNot(isExecutorDecommissioned)).map(_.toSet)
}
def hasExecutorsAliveOnHost(host: String): Boolean = synchronized {
- hostToExecutors.contains(host)
+ hostToExecutors.get(host)
+ .exists(executors => executors.exists(e => !isExecutorDecommissioned(e)))
}
def hasHostAliveOnRack(rack: String): Boolean = synchronized {
- hostsByRack.contains(rack)
+ hostsByRack.get(rack)
+ .exists(hosts => hosts.exists(h => !isHostDecommissioned(h)))
}
def isExecutorAlive(execId: String): Boolean = synchronized {
- executorIdToRunningTaskIds.contains(execId)
+ executorIdToRunningTaskIds.contains(execId) && !isExecutorDecommissioned(execId)
}
def isExecutorBusy(execId: String): Boolean = synchronized {
executorIdToRunningTaskIds.get(execId).exists(_.nonEmpty)
}
+ // exposed for test
+ protected def isExecutorDecommissioned(execId: String): Boolean =
+ getExecutorDecommissionInfo(execId).nonEmpty || isHostDecommissioned(executorIdToHost(execId))
Review comment:
Please consider removing `isHostDecommissioned` call here: When a host is decommissioned, all executors on that host will be marked decommissioned and would eventually end up in getExecutorDecommissionInfo(execId).nonEmpty.
I would really like the TaskSchedulerImpl to not make its own judgements of when an executor is decommissioned but rather let that be dictated by the CoarseGrainedSchedulerBackend/Cluster manager. Otherwise I am afraid that the two can get out of sync.
I understand the intention but I think it is a bit overzealous -- executors that are on decommissioned hosts shall eventually be marked as getExecutorDecommissionInfo.
I tested that your tests still pass even after it is removed.
##########
File path: core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala
##########
@@ -653,6 +652,33 @@ class TaskSetManagerSuite
assert(manager.resourceOffer("execA", "host1", ANY)._1.isDefined)
}
+ test("SPARK-32653: Decommissioned host/executor should not be used to calculate locality " +
+ "levels") {
+ sc = new SparkContext("local", "test")
+ sched = new FakeTaskScheduler(sc)
+ val backend = mock(classOf[SchedulerBackend])
+ doNothing().when(backend).reviveOffers()
+ sched.initialize(backend)
+
+ val exec0 = "exec0"
+ val exec1 = "exec1"
+ val host0 = "host0"
+ sched.addExecutor(exec0, host0)
+ sched.addExecutor(exec1, host0)
+
+ val taskSet = FakeTask.createTaskSet(2,
+ Seq(TaskLocation(host0, exec0)),
+ Seq(TaskLocation(host0, exec1)))
+ sched.submitTasks(taskSet)
+ val manager = sched.taskSetManagerForAttempt(0, 0).get
+
+ assert(manager.myLocalityLevels === Array(PROCESS_LOCAL, NODE_LOCAL, ANY))
+
+ sched.executorDecommission(exec0, ExecutorDecommissionInfo("test", true))
Review comment:
I think you should also call `sched.executorDecommission(exec1, ExecutorDecommissionInfo("test", true))` (ie for exec1), to mimic the production behavior which would always decommission all executors on a decommissioned host.
In addition, I think we should also test decommissioning when the entire host hasn't been taken down ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-677382297
**[Test build #127677 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127677/testReport)** for PR 29468 at commit [`65f6e05`](https://github.com/apache/spark/commit/65f6e0525b2d1472c1e6956587253e5ae6263676).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #29468:
URL: https://github.com/apache/spark/pull/29468#discussion_r472645226
##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##########
@@ -1062,25 +1062,36 @@ private[spark] class TaskSchedulerImpl(
}
def getExecutorsAliveOnHost(host: String): Option[Set[String]] = synchronized {
- hostToExecutors.get(host).map(_.toSet)
+ hostToExecutors.get(host).map(_.filterNot(isExecutorDecommissioned)).map(_.toSet)
}
def hasExecutorsAliveOnHost(host: String): Boolean = synchronized {
- hostToExecutors.contains(host)
+ hostToExecutors.get(host)
+ .exists(executors => executors.exists(e => !isExecutorDecommissioned(e)))
}
def hasHostAliveOnRack(rack: String): Boolean = synchronized {
- hostsByRack.contains(rack)
+ hostsByRack.get(rack)
Review comment:
`hostToExecutors` will also be used to handle executor removal and blacklisting stuff. In other words, a decommissioned executor could also be removed or blacklisted at the same time. So, remove the executor from `hostToExecutors` might also affect other places. I don't want to mix them together to make things complex.
Besides, similar to `executorIdToRunningTaskIds`, a decommissioned executor could also have running tasks on it. So we don't eagerly remove it from `executorIdToRunningTaskIds` too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-677395061
**[Test build #127680 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127680/testReport)** for PR 29468 at commit [`65f6e05`](https://github.com/apache/spark/commit/65f6e0525b2d1472c1e6956587253e5ae6263676).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-675677175
**[Test build #127592 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127592/testReport)** for PR 29468 at commit [`a4eaf5f`](https://github.com/apache/spark/commit/a4eaf5f5cd04ca3efc7de08b77224e35ca1fd5fe).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-677275274
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-677271793
**[Test build #127677 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127677/testReport)** for PR 29468 at commit [`65f6e05`](https://github.com/apache/spark/commit/65f6e0525b2d1472c1e6956587253e5ae6263676).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-677505209
**[Test build #127680 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127680/testReport)** for PR 29468 at commit [`65f6e05`](https://github.com/apache/spark/commit/65f6e0525b2d1472c1e6956587253e5ae6263676).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-676136425
**[Test build #127636 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127636/testReport)** for PR 29468 at commit [`aa2d5ba`](https://github.com/apache/spark/commit/aa2d5baf5983d80be491d3264b5ef80ad4c6b51b).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #29468:
URL: https://github.com/apache/spark/pull/29468#discussion_r472651414
##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##########
@@ -1062,25 +1062,36 @@ private[spark] class TaskSchedulerImpl(
}
def getExecutorsAliveOnHost(host: String): Option[Set[String]] = synchronized {
- hostToExecutors.get(host).map(_.toSet)
+ hostToExecutors.get(host).map(_.filterNot(isExecutorDecommissioned)).map(_.toSet)
Review comment:
hmm..I think this more like a problem of definition or something like that. In this PR, what I'm trying to do is to strengthen the definition of the active status for the executor or host. That is, for an active executor or host, it should not be decommissioned at the same time. This function is getting the active host, so we should filter out decommissioned ones. I think this's quite obvious. And this function is used by multiple tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-677279647
@agrawaldevesh Thanks for your commit. Changes make sense to me. I've merged it into this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-677602294
thanks, merging to master!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-675678163
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-676433269
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-677991943
thanks all!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-676430815
**[Test build #127648 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127648/testReport)** for PR 29468 at commit [`aa2d5ba`](https://github.com/apache/spark/commit/aa2d5baf5983d80be491d3264b5ef80ad4c6b51b).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-677395061
**[Test build #127680 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127680/testReport)** for PR 29468 at commit [`65f6e05`](https://github.com/apache/spark/commit/65f6e0525b2d1472c1e6956587253e5ae6263676).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-677275274
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-676138171
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127636/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] agrawaldevesh commented on a change in pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
agrawaldevesh commented on a change in pull request #29468:
URL: https://github.com/apache/spark/pull/29468#discussion_r472389478
##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##########
@@ -1062,25 +1062,36 @@ private[spark] class TaskSchedulerImpl(
}
def getExecutorsAliveOnHost(host: String): Option[Set[String]] = synchronized {
- hostToExecutors.get(host).map(_.toSet)
+ hostToExecutors.get(host).map(_.filterNot(isExecutorDecommissioned)).map(_.toSet)
Review comment:
I don't think this method needs to be changed: It only seems to be used when the HDFS split is cached in memory by Hadoop. Perhaps we should restrict to changing only the methods in the `computeValidLocalityLevels` call path ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-676191847
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-677382791
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-676138149
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29468:
URL: https://github.com/apache/spark/pull/29468#discussion_r472786561
##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##########
@@ -1062,25 +1062,36 @@ private[spark] class TaskSchedulerImpl(
}
def getExecutorsAliveOnHost(host: String): Option[Set[String]] = synchronized {
- hostToExecutors.get(host).map(_.toSet)
+ hostToExecutors.get(host).map(_.filterNot(isExecutorDecommissioned)).map(_.toSet)
}
def hasExecutorsAliveOnHost(host: String): Boolean = synchronized {
- hostToExecutors.contains(host)
+ hostToExecutors.get(host)
Review comment:
wrong indentation
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-675596852
**[Test build #127592 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127592/testReport)** for PR 29468 at commit [`a4eaf5f`](https://github.com/apache/spark/commit/a4eaf5f5cd04ca3efc7de08b77224e35ca1fd5fe).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-677396956
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-677382844
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127677/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-677396956
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-676138149
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29468:
URL: https://github.com/apache/spark/pull/29468#issuecomment-675915430
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org