You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@uniffle.apache.org by zu...@apache.org on 2026/07/14 07:48:22 UTC

(uniffle) branch master updated: [#2763] fix(spark)(gluten): use a 1.0 spill ratio for final clear to prevent potential data loss (#2764)

This is an automated email from the ASF dual-hosted git repository.

zuston pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/uniffle.git


The following commit(s) were added to refs/heads/master by this push:
     new d31ff12ed [#2763] fix(spark)(gluten): use a 1.0 spill ratio for final clear to prevent potential data loss (#2764)
d31ff12ed is described below

commit d31ff12eda6e782d38c20118532f4e86d8d9ff78
Author: Zhen Wang <64...@qq.com>
AuthorDate: Tue Jul 14 15:48:17 2026 +0800

    [#2763] fix(spark)(gluten): use a 1.0 spill ratio for final clear to prevent potential data loss (#2764)
    
    ### What changes were proposed in this pull request?
    
    Change the no-argument `WriteBufferManager.clear()` method used by the Gluten shuffle writer to call `clear(1.0)` instead of `clear(bufferSpillRatio)`.
    
    ### Why are the changes needed?
    
    A spill ratio of `1.0` should be used for this final call to ensure that all remaining buffered data is converted into shuffle blocks and flushed.
    
    Fix: #2763
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    No specific error was observed because we use the default `spillRatio` of `1.0`. This issue was identified while reviewing the relevant code. The existing unit tests already cover the `clear(1.0)` code path.
---
 .../main/java/org/apache/spark/shuffle/writer/WriteBufferManager.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/client-spark/common/src/main/java/org/apache/spark/shuffle/writer/WriteBufferManager.java b/client-spark/common/src/main/java/org/apache/spark/shuffle/writer/WriteBufferManager.java
index 280c6aefd..651b0b47e 100644
--- a/client-spark/common/src/main/java/org/apache/spark/shuffle/writer/WriteBufferManager.java
+++ b/client-spark/common/src/main/java/org/apache/spark/shuffle/writer/WriteBufferManager.java
@@ -371,7 +371,7 @@ public class WriteBufferManager extends MemoryConsumer {
 
   // Gluten needs this method.
   public synchronized List<ShuffleBlockInfo> clear() {
-    return clear(bufferSpillRatio);
+    return clear(1.0);
   }
 
   // transform all [partition, records] to [partition, ShuffleBlockInfo] and clear cache