You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@celeborn.apache.org by zh...@apache.org on 2023/07/10 03:29:55 UTC

[incubator-celeborn] branch main updated: [CELEBORN-779] Fix sorted file size summary overflow

This is an automated email from the ASF dual-hosted git repository.

zhouky pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-celeborn.git


The following commit(s) were added to refs/heads/main by this push:
     new a7ff28c26 [CELEBORN-779] Fix sorted file size summary overflow
a7ff28c26 is described below

commit a7ff28c26d9701ae3958513bdcc2b42df1a4c0af
Author: mingji <fe...@alibaba-inc.com>
AuthorDate: Mon Jul 10 11:29:47 2023 +0800

    [CELEBORN-779] Fix sorted file size summary overflow
    
    ### What changes were proposed in this pull request?
    To fix sorted file size summary counter overflow. Sorted file size summary can be larger than the integer's max value. It should be applied to branch-0.3, branch-0.2, and main branch.
    
    ### Why are the changes needed?
    This graph is incorrect.
    <img width="1500" alt="lQLPJx4a6A76xo7NBOHNC7iwLBx50tXnv5IEoQTQxMAMAA_3000_1249" src="https://github.com/apache/incubator-celeborn/assets/4150993/8d288087-64d7-4569-ba04-05bb1915118a">
    
    ### Does this PR introduce _any_ user-facing change?
    It will cause user-facing changes.  Celeborn worker's metric for sorted file summary is incorrect and this patch will correct it.
    
    ### How was this patch tested?
    UT.
    
    Closes #1694 from FMX/CELEBORN-779.
    
    Authored-by: mingji <fe...@alibaba-inc.com>
    Signed-off-by: zky.zhoukeyong <zk...@alibaba-inc.com>
---
 .../celeborn/service/deploy/worker/storage/PartitionFilesSorter.java  | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/worker/src/main/java/org/apache/celeborn/service/deploy/worker/storage/PartitionFilesSorter.java b/worker/src/main/java/org/apache/celeborn/service/deploy/worker/storage/PartitionFilesSorter.java
index 1d88069a8..016290513 100644
--- a/worker/src/main/java/org/apache/celeborn/service/deploy/worker/storage/PartitionFilesSorter.java
+++ b/worker/src/main/java/org/apache/celeborn/service/deploy/worker/storage/PartitionFilesSorter.java
@@ -154,8 +154,8 @@ public class PartitionFilesSorter extends ShuffleRecoverHelper {
     return sortedFileCount.get();
   }
 
-  public int getSortedSize() {
-    return (int) sortedFilesSize.get();
+  public long getSortedSize() {
+    return sortedFilesSize.get();
   }
 
   public FileInfo getSortedFileInfo(