You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "waitinfuture (via GitHub)" <gi...@apache.org> on 2023/06/19 07:03:12 UTC
[GitHub] [incubator-celeborn] waitinfuture opened a new pull request, #1603: [CELEBORN-692] WorkerStatusTracker::recordWorkerFailure should put WO…
waitinfuture opened a new pull request, #1603:
URL: https://github.com/apache/incubator-celeborn/pull/1603
…RKER_SHUTDOWN workers into shuttingWorkers
<!--
Thanks for sending a pull request! Here are some tips for you:
- Make sure the PR title start w/ a JIRA ticket, e.g. '[CELEBORN-XXXX] Your PR title ...'.
- Be sure to keep the PR description updated to reflect all changes.
- Please write your PR title to summarize what this PR proposes.
- If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
This PR put workers with WORKER_SHUTDOWN status into shuttingWorkers instead of blacklist.
### Why are the changes needed?
If WORKER_SHUTDOWN workers are put into blacklist, it will not trigger commit files, see ```CommitHandler::commitFiles```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manual test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on pull request #1603: [CELEBORN-692] WorkerStatusTracker::recordWorkerFailure should put WO…
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #1603:
URL: https://github.com/apache/incubator-celeborn/pull/1603#issuecomment-1596645486
wrong commit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] waitinfuture commented on a diff in pull request #1603: [CELEBORN-692] WorkerStatusTracker::recordWorkerFailure should put WO…
Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture commented on code in PR #1603:
URL: https://github.com/apache/incubator-celeborn/pull/1603#discussion_r1233620927
##########
client/src/main/scala/org/apache/celeborn/client/WorkerStatusTracker.scala:
##########
@@ -123,16 +123,20 @@ class WorkerStatusTracker(
|$blacklistMsg
""".stripMargin)
failedWorker.asScala.foreach { case (worker, (statusCode, registerTime)) =>
- if (!blacklist.containsKey(worker)) {
- blacklist.put(worker, (statusCode, registerTime))
+ if (StatusCode.WORKER_SHUTDOWN == statusCode) {
+ shuttingWorkers.add(worker)
} else {
- statusCode match {
- case StatusCode.WORKER_SHUTDOWN |
- StatusCode.NO_AVAILABLE_WORKING_DIR |
- StatusCode.RESERVE_SLOTS_FAILED |
- StatusCode.UNKNOWN_WORKER =>
- blacklist.put(worker, (statusCode, blacklist.get(worker)._2))
- case _ => // Not cover
+ if (!blacklist.containsKey(worker)) {
+ blacklist.put(worker, (statusCode, registerTime))
+ } else {
+ statusCode match {
+ case StatusCode.WORKER_SHUTDOWN |
Review Comment:
fixed, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] waitinfuture commented on pull request #1603: [CELEBORN-692] WorkerStatusTracker::recordWorkerFailure should put WO…
Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture commented on PR #1603:
URL: https://github.com/apache/incubator-celeborn/pull/1603#issuecomment-1596622993
cc @pan3793 @AngersZhuuuu @RexXiong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on pull request #1603: [CELEBORN-692] WorkerStatusTracker::recordWorkerFailure should put WO…
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #1603:
URL: https://github.com/apache/incubator-celeborn/pull/1603#issuecomment-1596646617
The code can be simplified by pattern match to avoid nested if-else
```
failedWorker.asScala.foreach {
case (worker, (StatusCode.WORKER_SHUTDOWN, _)) =>
shuttingWorkers.add(worker)
case (worker, (statusCode, registerTime)) if !blacklist.containsKey(worker) =>
blacklist.put(worker, (statusCode, registerTime))
case (worker, (statusCode, _))
if Array(
StatusCode.NO_AVAILABLE_WORKING_DIR,
StatusCode.RESERVE_SLOTS_FAILED,
StatusCode.UNKNOWN_WORKER).contains(statusCode) =>
blacklist.put(worker, (statusCode, blacklist.get(worker)._2))
case _ =>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1603: [CELEBORN-692] WorkerStatusTracker::recordWorkerFailure should put WO…
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1603:
URL: https://github.com/apache/incubator-celeborn/pull/1603#discussion_r1233614447
##########
client/src/main/scala/org/apache/celeborn/client/WorkerStatusTracker.scala:
##########
@@ -123,16 +123,20 @@ class WorkerStatusTracker(
|$blacklistMsg
""".stripMargin)
failedWorker.asScala.foreach { case (worker, (statusCode, registerTime)) =>
- if (!blacklist.containsKey(worker)) {
- blacklist.put(worker, (statusCode, registerTime))
+ if (StatusCode.WORKER_SHUTDOWN == statusCode) {
+ shuttingWorkers.add(worker)
} else {
- statusCode match {
- case StatusCode.WORKER_SHUTDOWN |
- StatusCode.NO_AVAILABLE_WORKING_DIR |
- StatusCode.RESERVE_SLOTS_FAILED |
- StatusCode.UNKNOWN_WORKER =>
- blacklist.put(worker, (statusCode, blacklist.get(worker)._2))
- case _ => // Not cover
+ if (!blacklist.containsKey(worker)) {
+ blacklist.put(worker, (statusCode, registerTime))
+ } else {
+ statusCode match {
+ case StatusCode.WORKER_SHUTDOWN |
Review Comment:
possible?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] waitinfuture commented on pull request #1603: [CELEBORN-692] WorkerStatusTracker::recordWorkerFailure should put WO…
Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture commented on PR #1603:
URL: https://github.com/apache/incubator-celeborn/pull/1603#issuecomment-1596661802
>
Refined, thanks :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] codecov[bot] commented on pull request #1603: [CELEBORN-692] WorkerStatusTracker::recordWorkerFailure should put WO…
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #1603:
URL: https://github.com/apache/incubator-celeborn/pull/1603#issuecomment-1596638966
## [Codecov](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1603?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#1603](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1603?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (79530cb) into [main](https://app.codecov.io/gh/apache/incubator-celeborn/commit/7d7107d607b853ab61b8299f83de2e3dacfa7e3c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (7d7107d) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 79530cb differs from pull request most recent head 52775ba. Consider uploading reports for the commit 52775ba to get more accurate results
```diff
@@ Coverage Diff @@
## main #1603 +/- ##
==========================================
- Coverage 45.81% 45.81% -0.00%
==========================================
Files 159 159
Lines 9950 9951 +1
Branches 970 970
==========================================
Hits 4558 4558
- Misses 5094 5095 +1
Partials 298 298
```
[see 6 files with indirect coverage changes](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1603/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 closed pull request #1603: [CELEBORN-692] WorkerStatusTracker should handle WORKER_SHUTDOWN properly
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed pull request #1603: [CELEBORN-692] WorkerStatusTracker should handle WORKER_SHUTDOWN properly
URL: https://github.com/apache/incubator-celeborn/pull/1603
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on pull request #1603: [CELEBORN-692] WorkerStatusTracker should handle WORKER_SHUTDOWN properly
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #1603:
URL: https://github.com/apache/incubator-celeborn/pull/1603#issuecomment-1596694716
Thanks, merged to main/0.3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org