(spark) branch branch-4.x updated: [SPARK-57921][CORE] Fix `spark.memory.offHeap.size` error message to require at least 1MiB

[do...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 960d2393bb30 [SPARK-57921][CORE] Fix `spark.memory.offHeap.size` error message to require at least 1MiB
960d2393bb30 is described below

commit 960d2393bb30370d8b69846d742f79aaf3f6e860
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Fri Jul 3 03:15:09 2026 -0700

    [SPARK-57921][CORE] Fix `spark.memory.offHeap.size` error message to require at least 1MiB
    
    ### What changes were proposed in this pull request?
    
    This PR aims to fix the error message of `Utils.checkOffHeapEnabled` to say that `spark.memory.offHeap.size` must be at least 1MiB, instead of `must be > 0`.
    
    ### Why are the changes needed?
    
    `Utils.executorOffHeapMemorySizeAsMb` converts `spark.memory.offHeap.size` to MiB with truncation before validation. So, a positive value less than 1MiB (e.g. `500000`) is truncated to `0` and rejected with a misleading message.
    
    ```
    $ bin/spark-shell -c spark.memory.offHeap.enabled=true -c spark.memory.offHeap.size=500000
    ...
    java.lang.IllegalArgumentException: requirement failed:
    spark.memory.offHeap.size must be > 0 when spark.memory.offHeap.enabled == true
    ```
    
    The user set a value greater than 0, but the error still says it `must be > 0`. Since all callers of `checkOffHeapEnabled` pass MiB-based values, the effective minimum is 1MiB. The check in `MemoryManager` which validates the raw byte value is untouched because `must be > 0` is accurate there.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No, this is an error message fix.
    
    ### How was this patch tested?
    
    Pass the CIs with the updated test case.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Fable 5
    
    Closes #56983 from dongjoon-hyun/SPARK-57921.
    
    Authored-by: Dongjoon Hyun <do...@apache.org>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
    (cherry picked from commit 5615098afd961b2fc345f487fbc5aa3f7c1b2cc6)
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 core/src/main/scala/org/apache/spark/util/Utils.scala      | 3 ++-
 core/src/test/scala/org/apache/spark/util/UtilsSuite.scala | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/core/src/main/scala/org/apache/spark/util/Utils.scala b/core/src/main/scala/org/apache/spark/util/Utils.scala
index 11dc885ca86b..2b03a068f132 100644
--- a/core/src/main/scala/org/apache/spark/util/Utils.scala
+++ b/core/src/main/scala/org/apache/spark/util/Utils.scala
@@ -3091,7 +3091,8 @@ private[spark] object Utils
   def checkOffHeapEnabled(sparkConf: ReadOnlySparkConf, offHeapSize: Long): Long = {
     if (sparkConf.get(MEMORY_OFFHEAP_ENABLED)) {
       require(offHeapSize > 0,
-        s"${MEMORY_OFFHEAP_SIZE.key} must be > 0 when ${MEMORY_OFFHEAP_ENABLED.key} == true")
+        s"${MEMORY_OFFHEAP_SIZE.key} must be at least 1MiB when " +
+          s"${MEMORY_OFFHEAP_ENABLED.key} == true")
       offHeapSize
     } else {
       0
diff --git a/core/src/test/scala/org/apache/spark/util/UtilsSuite.scala b/core/src/test/scala/org/apache/spark/util/UtilsSuite.scala
index 96e00db64bd4..e1e49a226735 100644
--- a/core/src/test/scala/org/apache/spark/util/UtilsSuite.scala
+++ b/core/src/test/scala/org/apache/spark/util/UtilsSuite.scala
@@ -1603,7 +1603,7 @@ class UtilsSuite extends SparkFunSuite with ResetSystemProperties {
     val sparkConf = new SparkConf()
       .set(MEMORY_OFFHEAP_ENABLED, true)
     val expected =
-      s"${MEMORY_OFFHEAP_SIZE.key} must be > 0 when ${MEMORY_OFFHEAP_ENABLED.key} == true"
+      s"${MEMORY_OFFHEAP_SIZE.key} must be at least 1MiB when ${MEMORY_OFFHEAP_ENABLED.key} == true"
     val message = intercept[IllegalArgumentException] {
       Utils.executorOffHeapMemorySizeAsMb(sparkConf)
     }.getMessage


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org