You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "AngersZhuuuu (via GitHub)" <gi...@apache.org> on 2023/06/06 09:49:03 UTC
[GitHub] [incubator-celeborn] AngersZhuuuu opened a new pull request, #1552: [CELEBORN-640][PERF] DataPushQueue should not keep waiting take tasks
AngersZhuuuu opened a new pull request, #1552:
URL: https://github.com/apache/incubator-celeborn/pull/1552
### What changes were proposed in this pull request?
In our prod meet many times of push queue stuck caused by PushState's status was not being removed.
Caused DataPushQueue to keep waiting for taking task.
Although have resolved some bugs, here we'd better add a max wait time for taking tasks since we already have the `PUSH_DATA_TIMEOUT` check method. If the target worker is really stuck, we can retry our task finally.
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] RexXiong commented on a diff in pull request #1552: [CELEBORN-640][PERF] DataPushQueue should not keep waiting take tasks
Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on code in PR #1552:
URL: https://github.com/apache/incubator-celeborn/pull/1552#discussion_r1222663385
##########
client/src/main/java/org/apache/celeborn/client/write/DataPushQueue.java:
##########
@@ -106,10 +110,17 @@ public ArrayList<PushTask> takePushTasks() throws IOException, InterruptedExcept
oldCapacity = maxInFlight - pushState.inflightPushes(loc.hostAndPushPort());
workerCapacity.put(loc.hostAndPushPort(), oldCapacity);
}
+ workerWaitAttempts.putIfAbsent(loc.hostAndPushPort(), new AtomicInteger(0));
if (oldCapacity > 0) {
iterator.remove();
tasks.add(task);
workerCapacity.put(loc.hostAndPushPort(), oldCapacity - 1);
+ } else if (workerWaitAttempts.get(loc.hostAndPushPort()).get()
+ >= takeTaskMaxWaitAttempts) {
Review Comment:
If the worker already very slowly, we better take only one task to push otherwise push task may block other worker push in next turn.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1552: [CELEBORN-640][PERF] DataPushQueue should not keep waiting take tasks
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1552:
URL: https://github.com/apache/incubator-celeborn/pull/1552#discussion_r1219412286
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -2777,14 +2778,22 @@ object CelebornConf extends Logging {
.intConf
.createWithDefault(8)
- val CLIENT_PUSH_TAKE_TASK_WAIT_TIME: ConfigEntry[Long] =
- buildConf("celeborn.client.push.takeTaskWaitTime")
+ val CLIENT_PUSH_TAKE_TASK_WAIT_INTERVAL: ConfigEntry[Long] =
+ buildConf("celeborn.client.push.takeTaskWaitInterval")
.categories("client")
.doc("Wait time if no task available to push to worker.")
.version("0.3.0")
.timeConf(TimeUnit.MILLISECONDS)
.createWithDefaultString("50ms")
+ val CLIENT_PUSH_TAKE_TASK_MAX_WAIT_ATTEMPTS: ConfigEntry[Int] =
+ buildConf("celeborn.client.push.takeTaskMaxWaitAttempts")
+ .categories("client")
+ .doc("Wait time if no task available to push to worker.")
Review Comment:
please update the doc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] codecov[bot] commented on pull request #1552: [CELEBORN-640][PERF] DataPushQueue should not keep waiting take tasks
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #1552:
URL: https://github.com/apache/incubator-celeborn/pull/1552#issuecomment-1578335481
## [Codecov](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1552?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#1552](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1552?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (110dafa) into [main](https://app.codecov.io/gh/apache/incubator-celeborn/commit/3c7d179e051dce1dc4cf3a559a6761eadeccd9a6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (3c7d179) will **increase** coverage by `0.20%`.
> The diff coverage is `80.00%`.
```diff
@@ Coverage Diff @@
## main #1552 +/- ##
==========================================
+ Coverage 46.21% 46.41% +0.20%
==========================================
Files 158 158
Lines 9958 9965 +7
Branches 967 967
==========================================
+ Hits 4601 4624 +23
+ Misses 5055 5043 -12
+ Partials 302 298 -4
```
| [Impacted Files](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1552?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...cala/org/apache/celeborn/common/CelebornConf.scala](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1552?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvY2VsZWJvcm4vY29tbW9uL0NlbGVib3JuQ29uZi5zY2FsYQ==) | `87.91% <80.00%> (-<0.01%)` | :arrow_down: |
... and [2 files with indirect coverage changes](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1552/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] AngersZhuuuu commented on a diff in pull request #1552: [CELEBORN-640][PERF] DataPushQueue should not keep waiting take tasks
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on code in PR #1552:
URL: https://github.com/apache/incubator-celeborn/pull/1552#discussion_r1221108287
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -2777,14 +2778,22 @@ object CelebornConf extends Logging {
.intConf
.createWithDefault(8)
- val CLIENT_PUSH_TAKE_TASK_WAIT_TIME: ConfigEntry[Long] =
- buildConf("celeborn.client.push.takeTaskWaitTime")
+ val CLIENT_PUSH_TAKE_TASK_WAIT_INTERVAL: ConfigEntry[Long] =
+ buildConf("celeborn.client.push.takeTaskWaitInterval")
.categories("client")
.doc("Wait time if no task available to push to worker.")
.version("0.3.0")
.timeConf(TimeUnit.MILLISECONDS)
.createWithDefaultString("50ms")
+ val CLIENT_PUSH_TAKE_TASK_MAX_WAIT_ATTEMPTS: ConfigEntry[Int] =
+ buildConf("celeborn.client.push.takeTaskMaxWaitAttempts")
+ .categories("client")
+ .doc("Wait time if no task available to push to worker.")
Review Comment:
DOne
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] AngersZhuuuu commented on a diff in pull request #1552: [CELEBORN-640][PERF] DataPushQueue should not keep waiting take tasks
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on code in PR #1552:
URL: https://github.com/apache/incubator-celeborn/pull/1552#discussion_r1219310949
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -2777,14 +2778,22 @@ object CelebornConf extends Logging {
.intConf
.createWithDefault(8)
- val CLIENT_PUSH_TAKE_TASK_WAIT_TIME: ConfigEntry[Long] =
- buildConf("celeborn.client.push.takeTaskWaitTime")
+ val CLIENT_PUSH_TAKE_TASK_WAIT_INTERVAL: ConfigEntry[Long] =
+ buildConf("celeborn.client.push.takeTaskWaitInterval")
Review Comment:
cc @pan3793 here changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] AngersZhuuuu commented on pull request #1552: [CELEBORN-640][PERF] DataPushQueue should not keep waiting take tasks
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on PR #1552:
URL: https://github.com/apache/incubator-celeborn/pull/1552#issuecomment-1581843456
ping @waitinfuture @RexXiong @FMX WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] AngersZhuuuu closed pull request #1552: [CELEBORN-640][WORKER] DataPushQueue should not keep waiting take tasks
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu closed pull request #1552: [CELEBORN-640][WORKER] DataPushQueue should not keep waiting take tasks
URL: https://github.com/apache/incubator-celeborn/pull/1552
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1552: [CELEBORN-640][PERF] DataPushQueue should not keep waiting take tasks
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1552:
URL: https://github.com/apache/incubator-celeborn/pull/1552#discussion_r1219327838
##########
docs/configuration/client.md:
##########
@@ -51,7 +51,8 @@ license: |
| celeborn.client.push.sort.randomizePartitionId.enabled | false | Whether to randomize partitionId in push sorter. If true, partitionId will be randomized when sort data to avoid skew when push to worker | 0.3.0 |
| celeborn.client.push.splitPartition.threads | 8 | Thread number to process shuffle split request in shuffle client. | 0.3.0 |
| celeborn.client.push.stageEnd.timeout | <value of celeborn.<module>.io.connectionTimeout> | Timeout for waiting StageEnd. During this process, there are `celeborn.client.requestCommitFiles.maxRetries` times for retry opportunities for committing filesand 1 times for releasing slots request. User can customize this value according to your setting. By default, the value is the max timeout value `celeborn.<module>.io.connectionTimeout`. | 0.3.0 |
-| celeborn.client.push.takeTaskWaitTime | 50ms | Wait time if no task available to push to worker. | 0.3.0 |
+| celeborn.client.push.takeTaskMaxWaitTimes | 1 | Wait time if no task available to push to worker. | 0.3.0 |
Review Comment:
"times" is confusing, use "attempts" instead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] AngersZhuuuu commented on pull request #1552: [CELEBORN-640][PERF] DataPushQueue should not keep waiting take tasks
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on PR #1552:
URL: https://github.com/apache/incubator-celeborn/pull/1552#issuecomment-1578315003
ping @pan3793 @waitinfuture @RexXiong @FMX
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] AngersZhuuuu commented on a diff in pull request #1552: [CELEBORN-640][PERF] DataPushQueue should not keep waiting take tasks
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on code in PR #1552:
URL: https://github.com/apache/incubator-celeborn/pull/1552#discussion_r1219365679
##########
docs/configuration/client.md:
##########
@@ -51,7 +51,8 @@ license: |
| celeborn.client.push.sort.randomizePartitionId.enabled | false | Whether to randomize partitionId in push sorter. If true, partitionId will be randomized when sort data to avoid skew when push to worker | 0.3.0 |
| celeborn.client.push.splitPartition.threads | 8 | Thread number to process shuffle split request in shuffle client. | 0.3.0 |
| celeborn.client.push.stageEnd.timeout | <value of celeborn.<module>.io.connectionTimeout> | Timeout for waiting StageEnd. During this process, there are `celeborn.client.requestCommitFiles.maxRetries` times for retry opportunities for committing filesand 1 times for releasing slots request. User can customize this value according to your setting. By default, the value is the max timeout value `celeborn.<module>.io.connectionTimeout`. | 0.3.0 |
-| celeborn.client.push.takeTaskWaitTime | 50ms | Wait time if no task available to push to worker. | 0.3.0 |
+| celeborn.client.push.takeTaskMaxWaitTimes | 1 | Wait time if no task available to push to worker. | 0.3.0 |
Review Comment:
DOne
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] AngersZhuuuu commented on pull request #1552: [CELEBORN-640][WORKER] DataPushQueue should not keep waiting take tasks
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on PR #1552:
URL: https://github.com/apache/incubator-celeborn/pull/1552#issuecomment-1583849549
@RexXiong Any more suggestion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org