You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "s0nskar (via GitHub)" <gi...@apache.org> on 2024/07/24 12:39:17 UTC

[PR] [CELEBORN-1519] Do not update estimated partition size if it is unchanged [celeborn]

s0nskar opened a new pull request, #2642:
URL: https://github.com/apache/celeborn/pull/2642

   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     - Make sure the PR title start w/ a JIRA ticket, e.g. '[CELEBORN-XXXX] Your PR title ...'.
     - Be sure to keep the PR description updated to reflect all changes.
     - Please write your PR title to summarize what this PR proposes.
     - If possible, provide a concise example to reproduce the issue for a faster review.
   -->
   
   ### What changes were proposed in this pull request?
   
   We will not update the estimated partition size if it is unchanged.
   
   ### Why are the changes needed?
   
   Celeborn currently triggers an workerinfo update even-though the estimated partition size is not changed. This leads to unnecessary logging and redundant worker info update operation.
   
   Example log -
   
   ```
   [master-partition-size-updater] WARN org.apache.celeborn.service.deploy.master.clustermeta.AbstractMetaManager - Celeborn cluster estimated partition size changed from 64.0 MiB to 64.0 MiB 
   ```
   
   ### Does this PR introduce _any_ user-facing change?
   
   NO
   
   
   ### How was this patch tested?
   Existing UT's
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1519] Do not update estimated partition size if it is unchanged [celeborn]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #2642:
URL: https://github.com/apache/celeborn/pull/2642#discussion_r1690738960


##########
master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/AbstractMetaManager.java:
##########
@@ -468,6 +468,12 @@ public void updatePartitionSize() {
     } else {
       estimatedPartitionSize = initialEstimatedPartitionSize;
     }
+
+    // Do not trigger update is estimated partition size value is unchanged
+    if (estimatedPartitionSize == oldEstimatedPartitionSize) {
+      return;

Review Comment:
   will this cause skip informing to new joined workers?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1519] Do not update estimated partition size if it is unchanged [celeborn]

Posted by "s0nskar (via GitHub)" <gi...@apache.org>.
s0nskar commented on code in PR #2642:
URL: https://github.com/apache/celeborn/pull/2642#discussion_r1691021625


##########
master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/AbstractMetaManager.java:
##########
@@ -468,6 +468,12 @@ public void updatePartitionSize() {
     } else {
       estimatedPartitionSize = initialEstimatedPartitionSize;
     }
+
+    // Do not trigger update is estimated partition size value is unchanged
+    if (estimatedPartitionSize == oldEstimatedPartitionSize) {
+      return;

Review Comment:
   For newly registered workers we update the estimated partition size on `RegisterWorker` event itself.
   
   https://github.com/apache/celeborn/blob/c0ca9523f9ca8db913a512521a9c3559d5ae19d5/master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/AbstractMetaManager.java#L256
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1519] Do not update estimated partition size if it is unchanged [celeborn]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed pull request #2642: [CELEBORN-1519] Do not update estimated partition size if it is unchanged
URL: https://github.com/apache/celeborn/pull/2642


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1519] Do not update estimated partition size if it is unchanged [celeborn]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #2642:
URL: https://github.com/apache/celeborn/pull/2642#discussion_r1691035438


##########
master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/AbstractMetaManager.java:
##########
@@ -468,6 +468,12 @@ public void updatePartitionSize() {
     } else {
       estimatedPartitionSize = initialEstimatedPartitionSize;
     }
+
+    // Do not trigger update is estimated partition size value is unchanged
+    if (estimatedPartitionSize == oldEstimatedPartitionSize) {
+      return;

Review Comment:
   Ha, it's okay then



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1519] Do not update estimated partition size if it is unchanged [celeborn]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #2642:
URL: https://github.com/apache/celeborn/pull/2642#issuecomment-2249755056

   Thanks, merged to main


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org