You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "AngersZhuuuu (via GitHub)" <gi...@apache.org> on 2023/06/06 09:49:03 UTC

[GitHub] [incubator-celeborn] AngersZhuuuu opened a new pull request, #1552: [CELEBORN-640][PERF] DataPushQueue should not keep waiting take tasks

AngersZhuuuu opened a new pull request, #1552:
URL: https://github.com/apache/incubator-celeborn/pull/1552

   ### What changes were proposed in this pull request?
   In our prod meet many times of push queue stuck caused by PushState's status was not being removed.
   Caused DataPushQueue to keep waiting for taking task.
   
   Although have resolved some bugs, here we'd better add a max wait time for taking tasks since we already have the `PUSH_DATA_TIMEOUT` check method. If the target worker is really stuck, we can retry our task finally.
   
   
   ### Why are the changes needed?
   
   
   
   ### Does this PR introduce _any_ user-facing change?
   
   
   
   ### How was this patch tested?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] RexXiong commented on a diff in pull request #1552: [CELEBORN-640][PERF] DataPushQueue should not keep waiting take tasks

Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on code in PR #1552:
URL: https://github.com/apache/incubator-celeborn/pull/1552#discussion_r1222663385


##########
client/src/main/java/org/apache/celeborn/client/write/DataPushQueue.java:
##########
@@ -106,10 +110,17 @@ public ArrayList<PushTask> takePushTasks() throws IOException, InterruptedExcept
               oldCapacity = maxInFlight - pushState.inflightPushes(loc.hostAndPushPort());
               workerCapacity.put(loc.hostAndPushPort(), oldCapacity);
             }
+            workerWaitAttempts.putIfAbsent(loc.hostAndPushPort(), new AtomicInteger(0));
             if (oldCapacity > 0) {
               iterator.remove();
               tasks.add(task);
               workerCapacity.put(loc.hostAndPushPort(), oldCapacity - 1);
+            } else if (workerWaitAttempts.get(loc.hostAndPushPort()).get()
+                >= takeTaskMaxWaitAttempts) {

Review Comment:
   If the worker already very slowly, we better take only one task to push  otherwise push task may block other worker push in next turn.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1552: [CELEBORN-640][PERF] DataPushQueue should not keep waiting take tasks

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1552:
URL: https://github.com/apache/incubator-celeborn/pull/1552#discussion_r1219412286


##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -2777,14 +2778,22 @@ object CelebornConf extends Logging {
       .intConf
       .createWithDefault(8)
 
-  val CLIENT_PUSH_TAKE_TASK_WAIT_TIME: ConfigEntry[Long] =
-    buildConf("celeborn.client.push.takeTaskWaitTime")
+  val CLIENT_PUSH_TAKE_TASK_WAIT_INTERVAL: ConfigEntry[Long] =
+    buildConf("celeborn.client.push.takeTaskWaitInterval")
       .categories("client")
       .doc("Wait time if no task available to push to worker.")
       .version("0.3.0")
       .timeConf(TimeUnit.MILLISECONDS)
       .createWithDefaultString("50ms")
 
+  val CLIENT_PUSH_TAKE_TASK_MAX_WAIT_ATTEMPTS: ConfigEntry[Int] =
+    buildConf("celeborn.client.push.takeTaskMaxWaitAttempts")
+      .categories("client")
+      .doc("Wait time if no task available to push to worker.")

Review Comment:
   please update the doc



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] codecov[bot] commented on pull request #1552: [CELEBORN-640][PERF] DataPushQueue should not keep waiting take tasks

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #1552:
URL: https://github.com/apache/incubator-celeborn/pull/1552#issuecomment-1578335481

   ## [Codecov](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1552?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   > Merging [#1552](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1552?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (110dafa) into [main](https://app.codecov.io/gh/apache/incubator-celeborn/commit/3c7d179e051dce1dc4cf3a559a6761eadeccd9a6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (3c7d179) will **increase** coverage by `0.20%`.
   > The diff coverage is `80.00%`.
   
   ```diff
   @@            Coverage Diff             @@
   ##             main    #1552      +/-   ##
   ==========================================
   + Coverage   46.21%   46.41%   +0.20%     
   ==========================================
     Files         158      158              
     Lines        9958     9965       +7     
     Branches      967      967              
   ==========================================
   + Hits         4601     4624      +23     
   + Misses       5055     5043      -12     
   + Partials      302      298       -4     
   ```
   
   
   | [Impacted Files](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1552?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [...cala/org/apache/celeborn/common/CelebornConf.scala](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1552?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvY2VsZWJvcm4vY29tbW9uL0NlbGVib3JuQ29uZi5zY2FsYQ==) | `87.91% <80.00%> (-<0.01%)` | :arrow_down: |
   
   ... and [2 files with indirect coverage changes](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1552/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] AngersZhuuuu commented on a diff in pull request #1552: [CELEBORN-640][PERF] DataPushQueue should not keep waiting take tasks

Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on code in PR #1552:
URL: https://github.com/apache/incubator-celeborn/pull/1552#discussion_r1221108287


##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -2777,14 +2778,22 @@ object CelebornConf extends Logging {
       .intConf
       .createWithDefault(8)
 
-  val CLIENT_PUSH_TAKE_TASK_WAIT_TIME: ConfigEntry[Long] =
-    buildConf("celeborn.client.push.takeTaskWaitTime")
+  val CLIENT_PUSH_TAKE_TASK_WAIT_INTERVAL: ConfigEntry[Long] =
+    buildConf("celeborn.client.push.takeTaskWaitInterval")
       .categories("client")
       .doc("Wait time if no task available to push to worker.")
       .version("0.3.0")
       .timeConf(TimeUnit.MILLISECONDS)
       .createWithDefaultString("50ms")
 
+  val CLIENT_PUSH_TAKE_TASK_MAX_WAIT_ATTEMPTS: ConfigEntry[Int] =
+    buildConf("celeborn.client.push.takeTaskMaxWaitAttempts")
+      .categories("client")
+      .doc("Wait time if no task available to push to worker.")

Review Comment:
   DOne



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] AngersZhuuuu commented on a diff in pull request #1552: [CELEBORN-640][PERF] DataPushQueue should not keep waiting take tasks

Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on code in PR #1552:
URL: https://github.com/apache/incubator-celeborn/pull/1552#discussion_r1219310949


##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -2777,14 +2778,22 @@ object CelebornConf extends Logging {
       .intConf
       .createWithDefault(8)
 
-  val CLIENT_PUSH_TAKE_TASK_WAIT_TIME: ConfigEntry[Long] =
-    buildConf("celeborn.client.push.takeTaskWaitTime")
+  val CLIENT_PUSH_TAKE_TASK_WAIT_INTERVAL: ConfigEntry[Long] =
+    buildConf("celeborn.client.push.takeTaskWaitInterval")

Review Comment:
   cc @pan3793 here changed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] AngersZhuuuu commented on pull request #1552: [CELEBORN-640][PERF] DataPushQueue should not keep waiting take tasks

Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on PR #1552:
URL: https://github.com/apache/incubator-celeborn/pull/1552#issuecomment-1581843456

   ping @waitinfuture @RexXiong @FMX WDYT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] AngersZhuuuu closed pull request #1552: [CELEBORN-640][WORKER] DataPushQueue should not keep waiting take tasks

Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu closed pull request #1552: [CELEBORN-640][WORKER] DataPushQueue should not keep waiting take tasks
URL: https://github.com/apache/incubator-celeborn/pull/1552


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1552: [CELEBORN-640][PERF] DataPushQueue should not keep waiting take tasks

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1552:
URL: https://github.com/apache/incubator-celeborn/pull/1552#discussion_r1219327838


##########
docs/configuration/client.md:
##########
@@ -51,7 +51,8 @@ license: |
 | celeborn.client.push.sort.randomizePartitionId.enabled | false | Whether to randomize partitionId in push sorter. If true, partitionId will be randomized when sort data to avoid skew when push to worker | 0.3.0 | 
 | celeborn.client.push.splitPartition.threads | 8 | Thread number to process shuffle split request in shuffle client. | 0.3.0 | 
 | celeborn.client.push.stageEnd.timeout | &lt;value of celeborn.&lt;module&gt;.io.connectionTimeout&gt; | Timeout for waiting StageEnd. During this process, there are `celeborn.client.requestCommitFiles.maxRetries` times for retry opportunities for committing filesand 1 times for releasing slots request. User can customize this value according to your setting. By default, the value is the max timeout value `celeborn.<module>.io.connectionTimeout`. | 0.3.0 | 
-| celeborn.client.push.takeTaskWaitTime | 50ms | Wait time if no task available to push to worker. | 0.3.0 | 
+| celeborn.client.push.takeTaskMaxWaitTimes | 1 | Wait time if no task available to push to worker. | 0.3.0 | 

Review Comment:
   "times" is confusing, use "attempts" instead



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] AngersZhuuuu commented on pull request #1552: [CELEBORN-640][PERF] DataPushQueue should not keep waiting take tasks

Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on PR #1552:
URL: https://github.com/apache/incubator-celeborn/pull/1552#issuecomment-1578315003

   ping @pan3793 @waitinfuture @RexXiong @FMX
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] AngersZhuuuu commented on a diff in pull request #1552: [CELEBORN-640][PERF] DataPushQueue should not keep waiting take tasks

Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on code in PR #1552:
URL: https://github.com/apache/incubator-celeborn/pull/1552#discussion_r1219365679


##########
docs/configuration/client.md:
##########
@@ -51,7 +51,8 @@ license: |
 | celeborn.client.push.sort.randomizePartitionId.enabled | false | Whether to randomize partitionId in push sorter. If true, partitionId will be randomized when sort data to avoid skew when push to worker | 0.3.0 | 
 | celeborn.client.push.splitPartition.threads | 8 | Thread number to process shuffle split request in shuffle client. | 0.3.0 | 
 | celeborn.client.push.stageEnd.timeout | &lt;value of celeborn.&lt;module&gt;.io.connectionTimeout&gt; | Timeout for waiting StageEnd. During this process, there are `celeborn.client.requestCommitFiles.maxRetries` times for retry opportunities for committing filesand 1 times for releasing slots request. User can customize this value according to your setting. By default, the value is the max timeout value `celeborn.<module>.io.connectionTimeout`. | 0.3.0 | 
-| celeborn.client.push.takeTaskWaitTime | 50ms | Wait time if no task available to push to worker. | 0.3.0 | 
+| celeborn.client.push.takeTaskMaxWaitTimes | 1 | Wait time if no task available to push to worker. | 0.3.0 | 

Review Comment:
   DOne



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] AngersZhuuuu commented on pull request #1552: [CELEBORN-640][WORKER] DataPushQueue should not keep waiting take tasks

Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on PR #1552:
URL: https://github.com/apache/incubator-celeborn/pull/1552#issuecomment-1583849549

   @RexXiong Any more suggestion?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org