You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "zhztheplayer (via GitHub)" <gi...@apache.org> on 2025/11/27 11:38:31 UTC

[PR] [VL] Fix OOM error is not thrown when global off-heap memory is running out for off-heap BHJ [incubator-gluten]

zhztheplayer opened a new pull request, #11208:
URL: https://github.com/apache/incubator-gluten/pull/11208

   A follow-up for https://github.com/apache/incubator-gluten/pull/11148 to make sure OOM is thrown when off-heap BHJ is running out of memory.
   
   Example:
   
   ```
   org.apache.gluten.memory.memtarget.ThrowOnOomMemoryTarget$OutOfMemoryException: Not enough spark off-heap execution memory. Acquired: 56.0 MiB, granted: 0.0 B. Try tweaking config option spark.memory.offHeap.size to get larger space to run this application (if spark.gluten.memory.dynamic.offHeap.sizing.enabled is not enabled). 
   Current config settings: 
   	spark.memory.offHeap.enabled=true
   	spark.gluten.memory.dynamic.offHeap.sizing.enabled=N/A
   	spark.gluten.memory.offHeap.size.in.bytes=N/A
   	spark.gluten.memory.task.offHeap.size.in.bytes=N/A
   	spark.gluten.memory.conservative.task.offHeap.size.in.bytes=N/A
   Global off-heap target stats: 
   	GlobalOffHeap.0: Current used bytes: 152.0 MiB, peak bytes: 152.0 MiB
   ...
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [VL] Fix OOM error not thrown when global off-heap memory is running out by off-heap BHJ [incubator-gluten]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #11208:
URL: https://github.com/apache/incubator-gluten/pull/11208#issuecomment-3588765535

   @jinchengchenghh @zjuwangg 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [VL] Fix OOM error not thrown when global off-heap memory is running out by off-heap BHJ [incubator-gluten]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #11208:
URL: https://github.com/apache/incubator-gluten/pull/11208#issuecomment-3589801300

   Thanks for the review @jinchengchenghh


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [VL] Fix OOM error not thrown when global off-heap memory is running out by off-heap BHJ [incubator-gluten]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #11208:
URL: https://github.com/apache/incubator-gluten/pull/11208#discussion_r2572013498


##########
backends-velox/src/test/scala/org/apache/spark/sql/execution/unsafe/UnsafeColumnarBuildSideRelationTest.scala:
##########
@@ -124,4 +167,14 @@ class UnsafeColumnarBuildSideRelationTest extends SharedSparkSession {
         sampleBytes.asInstanceOf[Array[AnyRef]]))
   }
 
+  test("Should throw OOM when off-heap memory is running out") {
+    // 500 MiB > 200 MiB so OOM should be thrown.
+    val relations = mutable.ListBuffer[UnsafeColumnarBuildSideRelation]()
+    assertThrows[OutOfMemoryException] {
+      for (i <- 0 until 10) {
+        relations += newUnsafeRelationWithHashMode(randomBytes(ByteUnit.MiB.toBytes(50).toInt))

Review Comment:
   Not really, we have to fill the byte array with non-zero data to guard the serde process.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [VL] Fix OOM error not thrown when global off-heap memory is running out by off-heap BHJ [incubator-gluten]

Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #11208:
URL: https://github.com/apache/incubator-gluten/pull/11208#discussion_r2571650193


##########
backends-velox/src/test/scala/org/apache/spark/sql/execution/unsafe/UnsafeColumnarBuildSideRelationTest.scala:
##########
@@ -124,4 +167,14 @@ class UnsafeColumnarBuildSideRelationTest extends SharedSparkSession {
         sampleBytes.asInstanceOf[Array[AnyRef]]))
   }
 
+  test("Should throw OOM when off-heap memory is running out") {
+    // 500 MiB > 200 MiB so OOM should be thrown.
+    val relations = mutable.ListBuffer[UnsafeColumnarBuildSideRelation]()
+    assertThrows[OutOfMemoryException] {
+      for (i <- 0 until 10) {
+        relations += newUnsafeRelationWithHashMode(randomBytes(ByteUnit.MiB.toBytes(50).toInt))

Review Comment:
   Do we need to use randomBytes, can we use a fixed empty bytes with size 50



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [GLUTEN-11169][VL] Fix OOM error not thrown when global off-heap memory is running out by off-heap BHJ [incubator-gluten]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer merged PR #11208:
URL: https://github.com/apache/incubator-gluten/pull/11208


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org