You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "boneanxs (via GitHub)" <gi...@apache.org> on 2023/02/09 03:23:13 UTC
[GitHub] [incubator-celeborn] boneanxs opened a new pull request, #1212: [CELEBORN-277] PushDataHandle callback could miss soft split status
boneanxs opened a new pull request, #1212:
URL: https://github.com/apache/incubator-celeborn/pull/1212
<!--
Thanks for sending a pull request! Here are some tips for you:
- Make sure the PR title start w/ a JIRA ticket, e.g. '[CELEBORN-XXXX] Your PR title ...'.
- Be sure to keep the PR description updated to reflect all changes.
- Please write your PR title to summarize what this PR proposes.
- If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
Return soft split status if soft split is true
### Why are the changes needed?
PushDataHandle callback could miss soft split status
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
Existing UT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] AngersZhuuuu commented on a diff in pull request #1212: [CELEBORN-277] PushDataHandle callback could miss soft split status
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on code in PR #1212:
URL: https://github.com/apache/incubator-celeborn/pull/1212#discussion_r1100946330
##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/PushDataHandler.scala:
##########
@@ -357,22 +357,27 @@ class PushDataHandler extends BaseMessageHandler with Logging {
// The codes here could be executed if
// 1. the client doesn't enable push data to the replica, the master worker could hit here
// 2. the client enables push data to the replica, and the replica worker could hit here
- Option(CongestionController.instance()) match {
- case Some(congestionController) =>
- if (congestionController.isUserCongested(fileWriter.getFileInfo.getUserIdentifier)) {
- if (isMaster) {
- callbackWithTimer.onSuccess(
- ByteBuffer.wrap(
- Array[Byte](StatusCode.PUSH_DATA_SUCCESS_MASTER_CONGESTED.getValue)))
+ if (softSplit.get()) {
+ callbackWithTimer.onSuccess(ByteBuffer.wrap(Array[Byte](StatusCode.SOFT_SPLIT.getValue)))
Review Comment:
Better to add some comment's to show if we return soft split , won't return congestion ect..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] waitinfuture merged pull request #1212: [CELEBORN-277] PushDataHandle callback could miss soft split status
Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture merged PR #1212:
URL: https://github.com/apache/incubator-celeborn/pull/1212
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] waitinfuture commented on a diff in pull request #1212: [CELEBORN-277] PushDataHandle callback could miss soft split status
Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture commented on code in PR #1212:
URL: https://github.com/apache/incubator-celeborn/pull/1212#discussion_r1101044554
##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/PushDataHandler.scala:
##########
@@ -285,6 +285,10 @@ class PushDataHandler extends BaseMessageHandler with Logging {
resp.flip()
callbackWithTimer.onSuccess(resp)
} else if (softSplit.get()) {
+ // TODO Currently if the worker is in soft split status, given the guess that the client
+ // will fast stop pushing data to the worker, we won't return congest status. But
+ // in the long term, especially if this issue could frequently happen, we may need to return
+ // congest&softSplit status together
Review Comment:
Agree with the comment, in the future we might encode multiple status in the response.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] codecov[bot] commented on pull request #1212: [CELEBORN-277] PushDataHandle callback could miss soft split status
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #1212:
URL: https://github.com/apache/incubator-celeborn/pull/1212#issuecomment-1423574083
# [Codecov](https://codecov.io/gh/apache/incubator-celeborn/pull/1212?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#1212](https://codecov.io/gh/apache/incubator-celeborn/pull/1212?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0521afa) into [main](https://codecov.io/gh/apache/incubator-celeborn/commit/cfe81969c943a072ad440b3d79d078866fd40552?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cfe8196) will **decrease** coverage by `0.04%`.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## main #1212 +/- ##
============================================
- Coverage 27.21% 27.17% -0.04%
+ Complexity 806 804 -2
============================================
Files 212 212
Lines 17953 17974 +21
Branches 1957 1964 +7
============================================
- Hits 4885 4882 -3
- Misses 12745 12768 +23
- Partials 323 324 +1
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-celeborn/pull/1212?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../scala/org/apache/celeborn/common/util/Utils.scala](https://codecov.io/gh/apache/incubator-celeborn/pull/1212?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvY2VsZWJvcm4vY29tbW9uL3V0aWwvVXRpbHMuc2NhbGE=) | `6.92% <0.00%> (-0.02%)` | :arrow_down: |
| [...leborn/service/deploy/worker/PushDataHandler.scala](https://codecov.io/gh/apache/incubator-celeborn/pull/1212?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-d29ya2VyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvY2VsZWJvcm4vc2VydmljZS9kZXBsb3kvd29ya2VyL1B1c2hEYXRhSGFuZGxlci5zY2FsYQ==) | `0.00% <0.00%> (ø)` | |
| [...ice/deploy/master/clustermeta/ha/HARaftServer.java](https://codecov.io/gh/apache/incubator-celeborn/pull/1212?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bWFzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9jZWxlYm9ybi9zZXJ2aWNlL2RlcGxveS9tYXN0ZXIvY2x1c3Rlcm1ldGEvaGEvSEFSYWZ0U2VydmVyLmphdmE=) | `76.58% <0.00%> (-1.35%)` | :arrow_down: |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] AngersZhuuuu commented on pull request #1212: [CELEBORN-277] PushDataHandle callback could miss soft split status
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on PR #1212:
URL: https://github.com/apache/incubator-celeborn/pull/1212#issuecomment-1423701907
ping @waitinfuture
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org