You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@uniffle.apache.org by zu...@apache.org on 2026/07/14 07:48:22 UTC
(uniffle) branch master updated: [#2763] fix(spark)(gluten): use a 1.0 spill ratio for final clear to prevent potential data loss (#2764)
This is an automated email from the ASF dual-hosted git repository.
zuston pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/uniffle.git
The following commit(s) were added to refs/heads/master by this push:
new d31ff12ed [#2763] fix(spark)(gluten): use a 1.0 spill ratio for final clear to prevent potential data loss (#2764)
d31ff12ed is described below
commit d31ff12eda6e782d38c20118532f4e86d8d9ff78
Author: Zhen Wang <64...@qq.com>
AuthorDate: Tue Jul 14 15:48:17 2026 +0800
[#2763] fix(spark)(gluten): use a 1.0 spill ratio for final clear to prevent potential data loss (#2764)
### What changes were proposed in this pull request?
Change the no-argument `WriteBufferManager.clear()` method used by the Gluten shuffle writer to call `clear(1.0)` instead of `clear(bufferSpillRatio)`.
### Why are the changes needed?
A spill ratio of `1.0` should be used for this final call to ensure that all remaining buffered data is converted into shuffle blocks and flushed.
Fix: #2763
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
No specific error was observed because we use the default `spillRatio` of `1.0`. This issue was identified while reviewing the relevant code. The existing unit tests already cover the `clear(1.0)` code path.
---
.../main/java/org/apache/spark/shuffle/writer/WriteBufferManager.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/client-spark/common/src/main/java/org/apache/spark/shuffle/writer/WriteBufferManager.java b/client-spark/common/src/main/java/org/apache/spark/shuffle/writer/WriteBufferManager.java
index 280c6aefd..651b0b47e 100644
--- a/client-spark/common/src/main/java/org/apache/spark/shuffle/writer/WriteBufferManager.java
+++ b/client-spark/common/src/main/java/org/apache/spark/shuffle/writer/WriteBufferManager.java
@@ -371,7 +371,7 @@ public class WriteBufferManager extends MemoryConsumer {
// Gluten needs this method.
public synchronized List<ShuffleBlockInfo> clear() {
- return clear(bufferSpillRatio);
+ return clear(1.0);
}
// transform all [partition, records] to [partition, ShuffleBlockInfo] and clear cache