You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhztheplayer (via GitHub)" <gi...@apache.org> on 2025/11/14 14:16:44 UTC

[PR] [SPARK-54354][SQL] Spark hangs when there's not enough JVM heap memory for broadcast hashed relation [spark]

zhztheplayer opened a new pull request, #53065:
URL: https://github.com/apache/spark/pull/53065

   ### What changes were proposed in this pull request?
   
   A fix to let Spark throw OOM rather than hang when there's not enough JVM heap memory for broadcast hashed relation. The fix is done by passing the current JVM's heap size rather than `Long.MaxValue / 2` to create the temporary `UnifiedMemoryManager` for broadcasting. This is an optimal practice because if the size we passed is too large, i.e., the current `Long.MaxValue / 2`, it will cause hanging; if the size is smaller than the current JVM heap size, the OOM might be thrown too early even when there's room in memory for the newly created hashed relation.
   
   Before:
   
   ```scala
   new UnifiedMemoryManager(
       new SparkConf().set(MEMORY_OFFHEAP_ENABLED.key, "false"),
       Long.MaxValue,
       Long.MaxValue / 2,
       1)
   ```
   
   After:
   
   ```scala
   UnifiedMemoryManager(
       new SparkConf().set(MEMORY_OFFHEAP_ENABLED.key, "false"), 1)
   ```
   
   ### Why are the changes needed?
   
   Report the error fast instead of hanging.
   
   ### Does this PR introduce _any_ user-facing change?
   
   In some scenarios where large unsafe hashed relations are allocated for broadcast hash join, user will see a meaningful OOM instead of hanging.
   
   Before:
   
   ```
   15:07:38.456 WARN org.apache.spark.memory.TaskMemoryManager: Failed to allocate a page (8589934592 bytes), try again.
   15:07:38.501 WARN org.apache.spark.memory.TaskMemoryManager: Failed to allocate a page (8589934592 bytes), try again.
   15:07:38.539 WARN org.apache.spark.memory.TaskMemoryManager: Failed to allocate a page (8589934592 bytes), try again.
   15:07:38.580 WARN org.apache.spark.memory.TaskMemoryManager: Failed to allocate a page (8589934592 bytes), try again.
   15:07:38.613 WARN org.apache.spark.memory.TaskMemoryManager: Failed to allocate a page (8589934592 bytes), try again.
   15:07:38.647 WARN org.apache.spark.memory.TaskMemoryManager: Failed to allocate a page (8589934592 bytes), try again.
   ...
   ```
   
   After:
   
   ```
   An exception or error caused a run to abort: [UNABLE_TO_ACQUIRE_MEMORY] Unable to acquire 8589934592 bytes of memory, got 7194909081. SQLSTATE: 53200 
   org.apache.spark.memory.SparkOutOfMemoryError: [UNABLE_TO_ACQUIRE_MEMORY] Unable to acquire 8589934592 bytes of memory, got 7194909081. SQLSTATE: 53200
   	at org.apache.spark.errors.SparkCoreErrors$.outOfMemoryError(SparkCoreErrors.scala:456)
   	at org.apache.spark.errors.SparkCoreErrors.outOfMemoryError(SparkCoreErrors.scala)
   	at org.apache.spark.memory.MemoryConsumer.throwOom(MemoryConsumer.java:157)
   	at org.apache.spark.memory.MemoryConsumer.allocateArray(MemoryConsumer.java:98)
   	at org.apache.spark.unsafe.map.BytesToBytesMap.allocate(BytesToBytesMap.java:868)
   	at org.apache.spark.unsafe.map.BytesToBytesMap.<init>(BytesToBytesMap.java:202)
   	at org.apache.spark.unsafe.map.BytesToBytesMap.<init>(BytesToBytesMap.java:209)
   	at org.apache.spark.sql.execution.joins.UnsafeHashedRelation$.apply(HashedRelation.scala:464)
   	at org.apache.spark.sql.execution.joins.HashedRelationSuite.$anonfun$new$90(HashedRelationSuite.scala:760)
   ```
   
   ### How was this patch tested?
   
   Added tests.
   
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   No.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54354][SQL] Fix Spark hanging when there's not enough JVM heap memory for broadcast hashed relation [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53065:
URL: https://github.com/apache/spark/pull/53065#discussion_r2538581314


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala:
##########
@@ -143,9 +143,8 @@ private[execution] object HashedRelation {
       new TaskMemoryManager(
         new UnifiedMemoryManager(
           new SparkConf().set(MEMORY_OFFHEAP_ENABLED.key, "false"),
-          Long.MaxValue,
-          Long.MaxValue / 2,
-          1),
+          Runtime.getRuntime.maxMemory,
+          Runtime.getRuntime.maxMemory / 2, 1),

Review Comment:
   ```suggestion
             Runtime.getRuntime.maxMemory / 2,
             1),
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54354][SQL] Fix Spark hanging when there's not enough JVM heap memory for broadcast hashed relation [spark]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #53065:
URL: https://github.com/apache/spark/pull/53065#issuecomment-3534317505

   @@HyukjinKwon @yaooqinn @dongjoon-hyun Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54354][SQL] Fix Spark hanging when there's not enough JVM heap memory for broadcast hashed relation [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #53065:
URL: https://github.com/apache/spark/pull/53065#discussion_r2550282554


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala:
##########
@@ -401,9 +401,8 @@ private[joins] class UnsafeHashedRelation(
     val taskMemoryManager = new TaskMemoryManager(
       new UnifiedMemoryManager(
         new SparkConf().set(MEMORY_OFFHEAP_ENABLED.key, "false"),
-        Long.MaxValue,
-        Long.MaxValue / 2,
-        1),
+        Runtime.getRuntime.maxMemory,
+        Runtime.getRuntime.maxMemory / 2, 1),

Review Comment:
   Hi, @zhztheplayer . Please split this into two line.
   
   ```
   - Runtime.getRuntime.maxMemory / 2, 1),
   + Runtime.getRuntime.maxMemory / 2,
     1),
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54354][SQL] Fix Spark hanging when there's not enough JVM heap memory for broadcast hashed relation [spark]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #53065:
URL: https://github.com/apache/spark/pull/53065#issuecomment-3563755837

   @dongjoon-hyun Done.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54354][SQL] Fix Spark hanging when there's not enough JVM heap memory for broadcast hashed relation [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53065:
URL: https://github.com/apache/spark/pull/53065#discussion_r2538587469


##########
sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala:
##########
@@ -753,4 +752,18 @@ class HashedRelationSuite extends SharedSparkSession {
       map.free()
     }
   }
+
+  test("UnsafeHashedRelation should throw OOM when there isn't enough memory") {
+    val relations = mutable.ArrayBuffer[HashedRelation]()
+    // We should finally see an OOM thrown since we are keeping allocating hashed relations.

Review Comment:
   This is a bad test, and will likely to break the CI process. Can we put it in the PR description as a manual test?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54354][SQL] Fix Spark hanging when there's not enough JVM heap memory for broadcast hashed relation [spark]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #53065:
URL: https://github.com/apache/spark/pull/53065#issuecomment-3558339916

   @cloud-fan Fixed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54354][SQL] Fix Spark hanging when there's not enough JVM heap memory for broadcast hashed relation [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53065:
URL: https://github.com/apache/spark/pull/53065#discussion_r2538583290


##########
sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala:
##########
@@ -753,4 +752,18 @@ class HashedRelationSuite extends SharedSparkSession {
       map.free()
     }
   }
+
+  test("UnsafeHashedRelation should throw OOM when there isn't enough memory") {

Review Comment:
   why did it hang before?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54354][SQL] Fix Spark hanging when there's not enough JVM heap memory for broadcast hashed relation [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #53065:
URL: https://github.com/apache/spark/pull/53065#issuecomment-3555647865

   can we fix the CI?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54354][SQL] Fix Spark hanging when there's not enough JVM heap memory for broadcast hashed relation [spark]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #53065:
URL: https://github.com/apache/spark/pull/53065#discussion_r2539087088


##########
sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala:
##########
@@ -753,4 +752,18 @@ class HashedRelationSuite extends SharedSparkSession {
       map.free()
     }
   }
+
+  test("UnsafeHashedRelation should throw OOM when there isn't enough memory") {

Review Comment:
   It's related to a logic introduced in PR https://github.com/apache/spark/pull/11095. In the PR, the following **"retry code"** is based on the assumption  that JVM heap memory could be slightly smaller than the specified on-heap size in UMM:
   
   https://github.com/davies/spark/blob/7ec7660381f3cd2047658f67b1882fccd83e95e5/core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java#L268-L276
   
   Because the code assumes the specified on-heap size in UMM is only finitely larger than the actual JVM heap size, so the call will return as soon as `current size + acquiredButNotUsed size` reaches the `specified heap size` limit.
   
   However, we set the on-heap size to an infinite value for broadcast hashed relation: https://github.com/apache/spark/blob/722bcc0f0d15245a39fae62c0c1c764e4b6a02f8/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala#L142-L150. So the **"retry code"** mentioned above will never end. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54354][SQL] Fix Spark hanging when there's not enough JVM heap memory for broadcast hashed relation [spark]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #53065:
URL: https://github.com/apache/spark/pull/53065#issuecomment-3563781864

   Thanks for the reviews, everyone.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54354][SQL] Fix Spark hanging when there's not enough JVM heap memory for broadcast hashed relation [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #53065: [SPARK-54354][SQL] Fix Spark hanging when there's not enough JVM heap memory for broadcast hashed relation
URL: https://github.com/apache/spark/pull/53065


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54354][SQL] Fix Spark hanging when there's not enough JVM heap memory for broadcast hashed relation [spark]

Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on code in PR #53065:
URL: https://github.com/apache/spark/pull/53065#discussion_r2540667849


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala:
##########
@@ -401,9 +400,8 @@ private[joins] class UnsafeHashedRelation(
     val taskMemoryManager = new TaskMemoryManager(
       new UnifiedMemoryManager(
         new SparkConf().set(MEMORY_OFFHEAP_ENABLED.key, "false"),
-        Long.MaxValue,
-        Long.MaxValue / 2,
-        1),
+        Runtime.getRuntime.maxMemory,
+        Runtime.getRuntime.maxMemory / 2, 1),
       0)
 
     val pageSizeBytes = Option(SparkEnv.get).map(_.memoryManager.pageSizeBytes)

Review Comment:
   Do you know if the per-JVM memory manager here can be used?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54354][SQL] Fix Spark hanging when there's not enough JVM heap memory for broadcast hashed relation [spark]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #53065:
URL: https://github.com/apache/spark/pull/53065#issuecomment-3542696331

   cc @cloud-fan


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54354][SQL] Fix Spark hanging when there's not enough JVM heap memory for broadcast hashed relation [spark]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #53065:
URL: https://github.com/apache/spark/pull/53065#discussion_r2539087088


##########
sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala:
##########
@@ -753,4 +752,18 @@ class HashedRelationSuite extends SharedSparkSession {
       map.free()
     }
   }
+
+  test("UnsafeHashedRelation should throw OOM when there isn't enough memory") {

Review Comment:
   It's related to a logic introduced in PR https://github.com/apache/spark/pull/11095. In the PR, the following **"retry code"** is based on the assumption  that JVM heap memory could be slightly smaller than the specified on-heap size in UMM:
   
   https://github.com/davies/spark/blob/7ec7660381f3cd2047658f67b1882fccd83e95e5/core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java#L268-L276
   
   Because the code assumes the specified on-heap size in UMM is only finitely larger than the actual JVM heap size, so the call will return as soon as `current size + acquiredButNotUsed size` reaches the `specified heap size` limit.
   
   However, we set the on-heap size to an infinite value for broadcast hashed relation: https://github.com/apache/spark/blob/722bcc0f0d15245a39fae62c0c1c764e4b6a02f8/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala#L142-L150. So the **"retry code"** mentioned above will never end until an OOM error or a stack overflow error. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54354][SQL] Fix Spark hanging when there's not enough JVM heap memory for broadcast hashed relation [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #53065:
URL: https://github.com/apache/spark/pull/53065#issuecomment-3563766324

   Merged to master/4.1 for Apache Spark 4.1.0 to fix the hang issues. Thank you, @zhztheplayer , @cloud-fan , @yaooqinn .


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54354][SQL] Fix Spark hanging when there's not enough JVM heap memory for broadcast hashed relation [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #53065:
URL: https://github.com/apache/spark/pull/53065#discussion_r2550283174


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala:
##########
@@ -576,9 +575,8 @@ private[execution] final class LongToUnsafeRowMap(
       new TaskMemoryManager(
         new UnifiedMemoryManager(
           new SparkConf().set(MEMORY_OFFHEAP_ENABLED.key, "false"),
-          Long.MaxValue,
-          Long.MaxValue / 2,
-          1),
+          Runtime.getRuntime.maxMemory,
+          Runtime.getRuntime.maxMemory / 2, 1),

Review Comment:
   ditto.



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala:
##########
@@ -42,9 +42,8 @@ import org.apache.spark.util.collection.CompactBuffer
 class HashedRelationSuite extends SharedSparkSession {
   val umm = new UnifiedMemoryManager(
     new SparkConf().set(MEMORY_OFFHEAP_ENABLED.key, "false"),
-    Long.MaxValue,
-    Long.MaxValue / 2,
-    1)
+    Runtime.getRuntime.maxMemory,
+    Runtime.getRuntime.maxMemory / 2, 1)

Review Comment:
   ditto.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54354][SQL] Fix Spark hanging when there's not enough JVM heap memory for broadcast hashed relation [spark]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #53065:
URL: https://github.com/apache/spark/pull/53065#discussion_r2539046513


##########
sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala:
##########
@@ -753,4 +752,18 @@ class HashedRelationSuite extends SharedSparkSession {
       map.free()
     }
   }
+
+  test("UnsafeHashedRelation should throw OOM when there isn't enough memory") {
+    val relations = mutable.ArrayBuffer[HashedRelation]()
+    // We should finally see an OOM thrown since we are keeping allocating hashed relations.

Review Comment:
   Hi @cloud-fan, thanks for reviewing. 
   
   > This is a bad test, and will likely to break the CI process.
   
   If you meant the OOM error could break the CI, I think we already rely on the similar logic in the production code: https://github.com/apache/spark/blob/dce992b7708bc0d0f365aa94a40991a225696145/core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java#L390-L403. So I thought it is benign to catch them in testing? 
   
   Or is there anything else you are concerned about?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54354][SQL] Fix Spark hanging when there's not enough JVM heap memory for broadcast hashed relation [spark]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #53065:
URL: https://github.com/apache/spark/pull/53065#discussion_r2541374942


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala:
##########
@@ -401,9 +400,8 @@ private[joins] class UnsafeHashedRelation(
     val taskMemoryManager = new TaskMemoryManager(
       new UnifiedMemoryManager(
         new SparkConf().set(MEMORY_OFFHEAP_ENABLED.key, "false"),
-        Long.MaxValue,
-        Long.MaxValue / 2,
-        1),
+        Runtime.getRuntime.maxMemory,
+        Runtime.getRuntime.maxMemory / 2, 1),
       0)
 
     val pageSizeBytes = Option(SparkEnv.get).map(_.memoryManager.pageSizeBytes)

Review Comment:
   Not right now but I think it's the final target. We cannot directly modify the memory manager here, because currently Spark has to stick BHJ's memory allocation on the JVM heap. The per-JVM memory manager may be using off-heap mode.
   
   https://github.com/apache/spark/pull/52817 will improve SHJ to make sure it follows the per-JVM memory manager's memory mode, but we need a separate solution for BHJ in the future (which relies on the `read` code path marked here).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54354][SQL] Fix Spark hanging when there's not enough JVM heap memory for broadcast hashed relation [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53065:
URL: https://github.com/apache/spark/pull/53065#discussion_r2540276981


##########
sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala:
##########
@@ -753,4 +752,18 @@ class HashedRelationSuite extends SharedSparkSession {
       map.free()
     }
   }
+
+  test("UnsafeHashedRelation should throw OOM when there isn't enough memory") {
+    val relations = mutable.ArrayBuffer[HashedRelation]()
+    // We should finally see an OOM thrown since we are keeping allocating hashed relations.

Review Comment:
   It's ok to test with "managed" OOM that is thrown by us, but not a real OOM that destabilize the CI service. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54354][SQL] Fix Spark hanging when there's not enough JVM heap memory for broadcast hashed relation [spark]

Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on code in PR #53065:
URL: https://github.com/apache/spark/pull/53065#discussion_r2540667849


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala:
##########
@@ -401,9 +400,8 @@ private[joins] class UnsafeHashedRelation(
     val taskMemoryManager = new TaskMemoryManager(
       new UnifiedMemoryManager(
         new SparkConf().set(MEMORY_OFFHEAP_ENABLED.key, "false"),
-        Long.MaxValue,
-        Long.MaxValue / 2,
-        1),
+        Runtime.getRuntime.maxMemory,
+        Runtime.getRuntime.maxMemory / 2, 1),
       0)
 
     val pageSizeBytes = Option(SparkEnv.get).map(_.memoryManager.pageSizeBytes)

Review Comment:
   Do you know if the per-JVM memory manager can be used?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org