[I] Spark memory configs read with an accessor that does not match their declared unit [gluten]

["LuciferYang (via GitHub)" <gi...@apache.org>]

LuciferYang opened a new issue, #12676:
URL: https://github.com/apache/gluten/issues/12676

   ### Backend
   
   VL (Velox). The code is backend-agnostic and lives in `gluten-core` and `gluten-substrait`.
   
   ### Bug description
   
   Three Spark memory configurations are declared with a unit but read through an accessor that assumes a different one, so the off-heap budgets Gluten derives from them are wrong.
   
   1. `spark.executor.memory` is declared `bytesConf(ByteUnit.MiB)`, so a value without a suffix means MiB. `SparkResourceUtil.getExecutorMemorySize` read it with `getSizeAsBytes`, which reads a suffix-less value as bytes. `spark.executor.memory=8192` gave 8192 bytes instead of 8 GiB, and the dynamic off-heap sizing path derived its budget from that.
   
   2. `spark.executor.minMemoryOverhead` is a size string, MiB unless suffixed. `getMemoryOverheadSize` read it with `conf.getLong`, which is `String.toLong` and cannot parse `512m`. On a value Spark itself accepts, that threw `NumberFormatException` from `VeloxListenerApi#onDriverStart` and aborted driver startup.
   
   3. A `ResourceProfile` records executor memory amounts in MiB, while `spark.gluten.memory.offHeap.size.in.bytes` and `spark.gluten.memory.task.offHeap.size.in.bytes` are declared `bytesConf(ByteUnit.BYTE)`. `GlutenAutoAdjustStageResourceProfile#updateResourceSetting` wrote the profile amount verbatim, so a 20 GiB off-heap budget became 20480 bytes.
   
   `ByteUnit.MiB.toBytes` rejects a negative input but wraps past 2^43 MiB without raising. `getMemoryOverheadSize` ended in `toBytes`, so `spark.executor.minMemoryOverhead=9000000t` produced a negative overhead budget. `VeloxListenerApi#onDriverStart` writes that into `spark.gluten.memory.memoryOverhead.size.in.bytes`, and `VeloxBackend.cc` multiplies it by 0.75 to size the Velox global allocator.
   
   `updateResourceSetting` computes the task slot count itself rather than calling `SparkResourceUtil.getTaskSlots`, and the two disagree in local mode: `getTaskSlots` resolves `local[8]` to 8 slots, while a profile reports `spark.executor.cores`, 1 by default. While the total was also 2^20 times too small the two errors cancelled, so the divergence had no visible effect. Correct the total and it becomes an 8x over-provision, with each of 8 concurrent tasks sized as if it owned the whole budget. The same expression has no positivity check and no floor, so a profile with `CORES=1` and `spark.task.cpus=2` throws `ArithmeticException: / by zero` on every query.
   
   ### Gluten version
   
   main (1.8.0-SNAPSHOT)
   
   ### Spark version
   
   Version-agnostic (applies to spark-3.3 / 3.4 / 3.5 / 4.0 / 4.1).
   
   ### Spark configurations
   
   The divide-by-zero needs `spark.gluten.auto.adjustStageResource.enabled=true`, `spark.sql.adaptive.enabled=true`, `spark.memory.offHeap.enabled=true`, `spark.task.cpus=2`, and master `local[8]`.
   
   The unit errors need `spark.memory.offHeap.enabled=true` plus one of `spark.executor.memory=8192`, `spark.executor.minMemoryOverhead=512m`, or `spark.memory.offHeap.size=20g`.
   
   ### System information
   
   Not applicable. The affected code reads configuration in `gluten-core/src/main/scala/org/apache/spark/util/SparkResourceUtil.scala` and `gluten-substrait/src/main/scala/org/apache/spark/sql/execution/GlutenAutoAdjustStageResourceProfile.scala`, and does not depend on OS or hardware.
   
   ### Relevant logs
   
   ```text
   java.lang.NumberFormatException: Illegal value for config key spark.executor.minMemoryOverhead: ... Failed to parse byte string: 512m
     at org.apache.gluten.backendsapi.velox.VeloxListenerApi.onDriverStart(VeloxListenerApi.scala:116)
   
   java.lang.ArithmeticException: / by zero
     at org.apache.spark.sql.execution.GlutenAutoAdjustStageResourceProfile$.updateResourceSetting(GlutenAutoAdjustStageResourceProfile.scala:206)
   ```
   
   ### Fix direction
   
   Read each config with the accessor that matches its declared unit. Put the MiB-to-byte conversion behind one helper in `SparkResourceUtil` that guards both the sign and the magnitude, and route all three boundaries through it. In `updateResourceSetting`, defer to `SparkResourceUtil.getTaskSlots` in local mode, where the profile does not reflect the thread count, and give the other branch the positivity check and one-slot floor that `getTaskSlots` already has. Read the exact conf bytes when the profile is the unmodified default, since reading through the profile truncates to MiB.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org

Re: [I] Spark memory configs read with an accessor that does not match their declared unit [gluten]

["jackylee-ch (via GitHub)" <gi...@apache.org>]

jackylee-ch closed issue #12676: Spark memory configs read with an accessor that does not match their declared unit
URL: https://github.com/apache/gluten/issues/12676


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org