You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by ag...@apache.org on 2026/04/14 14:20:14 UTC
(datafusion-comet) branch branch-0.14 updated: fix: use min instead of max when capping write buffer size to Int range (#3914) (#3936)
This is an automated email from the ASF dual-hosted git repository.
agrove pushed a commit to branch branch-0.14
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/branch-0.14 by this push:
new 3a730de4d fix: use min instead of max when capping write buffer size to Int range (#3914) (#3936)
3a730de4d is described below
commit 3a730de4d3091942da65f33303d10d38b3095b33
Author: Andy Grove <ag...@apache.org>
AuthorDate: Tue Apr 14 08:20:09 2026 -0600
fix: use min instead of max when capping write buffer size to Int range (#3914) (#3936)
COMET_SHUFFLE_WRITE_BUFFER_SIZE is a Long (bytesConf) but the protobuf
field is int32, so the value must be capped at Int.MaxValue. The code
used .max(Int.MaxValue) which always returns Int.MaxValue (~2GB)
regardless of the configured value. Should be .min(Int.MaxValue) to
preserve smaller values while capping at the Int range.
---
.../spark/sql/comet/execution/shuffle/CometNativeShuffleWriter.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometNativeShuffleWriter.scala b/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometNativeShuffleWriter.scala
index 3fc222bd1..d704d3fd8 100644
--- a/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometNativeShuffleWriter.scala
+++ b/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometNativeShuffleWriter.scala
@@ -191,7 +191,7 @@ class CometNativeShuffleWriter[K, V](
shuffleWriterBuilder.setCompressionLevel(
CometConf.COMET_EXEC_SHUFFLE_COMPRESSION_ZSTD_LEVEL.get)
shuffleWriterBuilder.setWriteBufferSize(
- CometConf.COMET_SHUFFLE_WRITE_BUFFER_SIZE.get().max(Int.MaxValue).toInt)
+ CometConf.COMET_SHUFFLE_WRITE_BUFFER_SIZE.get().min(Int.MaxValue).toInt)
outputPartitioning match {
case p if isSinglePartitioning(p) =>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org