You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhztheplayer (via GitHub)" <gi...@apache.org> on 2025/10/31 12:24:49 UTC

[PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]

zhztheplayer opened a new pull request, #52817:
URL: https://github.com/apache/spark/pull/52817

   ### What changes were proposed in this pull request?
   
   The PR adds off-heap memory mode support for LongHashedRelation.
   
   ### Why are the changes needed?
   
   To avoid on-heap OOMs when user sets `spark.memory.offHeap.enabled=true`, and configures a comparatively small JVM heap size.
   
   ### Does this PR introduce _any_ user-facing change?
   
   When `spark.memory.offHeap.enabled=true` is set:
   - Required off-heap memory size may increase
   - Required on-heap memory size may decrease
   
   ### How was this patch tested?
   
   WIP
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   No.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52817:
URL: https://github.com/apache/spark/pull/52817#discussion_r2555447358


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala:
##########
@@ -816,35 +809,34 @@ private[execution] final class LongToUnsafeRowMap(
     } else {
       // there are some values for this key, put the address in the front of them.
       val pointer = toOffset(address) + toSize(address)
-      Platform.putLong(page, pointer, array(pos + 1))
+      Platform.putLong(page.getBaseObject, page.getBaseOffset + pointer, array(pos + 1))
       array(pos + 1) = address
     }
   }
 
   private def grow(inputRowSize: Int): Unit = {
     // There is 8 bytes for the pointer to next value
-    val neededNumWords = (cursor - Platform.LONG_ARRAY_OFFSET + 8 + inputRowSize + 7) / 8
-    if (neededNumWords > page.length) {
+    val usedBytes = cursor - page.getBaseOffset
+    val neededNumWords = (usedBytes + 8 + inputRowSize + 7) / 8
+    if (neededNumWords > page.size() / 8) {
       if (neededNumWords > (1 << 30)) {
         throw QueryExecutionErrors.cannotBuildHashedRelationLargerThan8GError()
       }
-      val newNumWords = math.max(neededNumWords, math.min(page.length * 2, 1 << 30))
-      ensureAcquireMemory(newNumWords * 8L)
-      val newPage = new Array[Long](newNumWords.toInt)
-      Platform.copyMemory(page, Platform.LONG_ARRAY_OFFSET, newPage, Platform.LONG_ARRAY_OFFSET,
-        cursor - Platform.LONG_ARRAY_OFFSET)
-      val used = page.length
+      val newNumWords = math.max(neededNumWords, math.min(page.size() / 8 * 2, 1 << 30))
+      val newPage = allocatePage(newNumWords.toInt * 8)
+      Platform.copyMemory(page.getBaseObject, page.getBaseOffset, newPage.getBaseObject,
+        newPage.getBaseOffset, usedBytes)
+      freePage(page)
       page = newPage
-      freeMemory(used * 8L)
+      cursor = page.getBaseOffset + usedBytes
     }
   }
 
   private def growArray(): Unit = {
     var old_array = array
     val n = array.length
     numKeys = 0
-    ensureAcquireMemory(n * 2 * 8L)
-    array = new Array[Long](n * 2)
+    array = new UnsafeLongArray(n * 2)

Review Comment:
   Does `UnsafeLongArray` allocate memory from the memory manager?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #52817:
URL: https://github.com/apache/spark/pull/52817#issuecomment-3513284335

   > e.g., sometimes it appears JIT to be quite smarter than using direct off heap memory
   
   I actually didn't expect the off-heap relation can be faster - it's more a work to make sure Spark can work with a relatively smaller heap under off-heap memory mode, so we can prevent heap OOMs. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #52817: [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation
URL: https://github.com/apache/spark/pull/52817


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #52817:
URL: https://github.com/apache/spark/pull/52817#issuecomment-3513191628

   e.g., sometimes it appears JIT to be quite smarter than using direct off heap memory


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #52817:
URL: https://github.com/apache/spark/pull/52817#issuecomment-3573896164

   thanks, merging to master!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation to avoid JVM heap OOMs under off-heap memory mode [spark]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #52817:
URL: https://github.com/apache/spark/pull/52817#issuecomment-3517111427

   @HyukjinKwon, It's interesting that, after the change, the off-heap mode relation looks meaningfully faster than on-heap relation in the benchmark. I will update the benchmark result inline and in PR description.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #52817:
URL: https://github.com/apache/spark/pull/52817#discussion_r2558129392


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala:
##########
@@ -985,6 +975,26 @@ private[execution] final class LongToUnsafeRowMap(
   override def read(kryo: Kryo, in: Input): Unit = {
     read(() => in.readBoolean(), () => in.readLong(), in.readBytes)
   }
+
+  private class UnsafeLongArray(val length: Int) {

Review Comment:
   It's a bit more code changes using `LongArray` - but yes the functinality is the same.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #52817:
URL: https://github.com/apache/spark/pull/52817#discussion_r2557248391


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala:
##########
@@ -816,35 +809,34 @@ private[execution] final class LongToUnsafeRowMap(
     } else {
       // there are some values for this key, put the address in the front of them.
       val pointer = toOffset(address) + toSize(address)
-      Platform.putLong(page, pointer, array(pos + 1))
+      Platform.putLong(page.getBaseObject, page.getBaseOffset + pointer, array(pos + 1))
       array(pos + 1) = address
     }
   }
 
   private def grow(inputRowSize: Int): Unit = {
     // There is 8 bytes for the pointer to next value
-    val neededNumWords = (cursor - Platform.LONG_ARRAY_OFFSET + 8 + inputRowSize + 7) / 8
-    if (neededNumWords > page.length) {
+    val usedBytes = cursor - page.getBaseOffset
+    val neededNumWords = (usedBytes + 8 + inputRowSize + 7) / 8
+    if (neededNumWords > page.size() / 8) {
       if (neededNumWords > (1 << 30)) {
         throw QueryExecutionErrors.cannotBuildHashedRelationLargerThan8GError()
       }
-      val newNumWords = math.max(neededNumWords, math.min(page.length * 2, 1 << 30))
-      ensureAcquireMemory(newNumWords * 8L)
-      val newPage = new Array[Long](newNumWords.toInt)
-      Platform.copyMemory(page, Platform.LONG_ARRAY_OFFSET, newPage, Platform.LONG_ARRAY_OFFSET,
-        cursor - Platform.LONG_ARRAY_OFFSET)
-      val used = page.length
+      val newNumWords = math.max(neededNumWords, math.min(page.size() / 8 * 2, 1 << 30))
+      val newPage = allocatePage(newNumWords.toInt * 8)
+      Platform.copyMemory(page.getBaseObject, page.getBaseOffset, newPage.getBaseObject,
+        newPage.getBaseOffset, usedBytes)
+      freePage(page)
       page = newPage
-      freeMemory(used * 8L)
+      cursor = page.getBaseOffset + usedBytes
     }
   }
 
   private def growArray(): Unit = {
     var old_array = array
     val n = array.length
     numKeys = 0
-    ensureAcquireMemory(n * 2 * 8L)
-    array = new Array[Long](n * 2)
+    array = new UnsafeLongArray(n * 2)

Review Comment:
   Yes, it calls the memory manager in its constructor.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #52817:
URL: https://github.com/apache/spark/pull/52817#issuecomment-3513268573

   @HyukjinKwon Thank you for the quick response.
   
   I benchmarked using the existing HashedRelationMetricsBenchmark.
   
   ### 500K Rows
   
   Before:
   
   ```
   OpenJDK 64-Bit Server VM 17.0.16+8-Ubuntu-0ubuntu124.04.1 on Linux 6.14.0-33-generic
   18:40:31.460 ERROR org.apache.spark.util.Utils: Process List(/usr/bin/grep, -m, 1, model name, /proc/cpuinfo) exited with code 1: 
   
   Unknown processor
   LongToUnsafeRowMap metrics:               Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
   ------------------------------------------------------------------------------------------------------------------------
   LongToUnsafeRowMap                                   55             63           4          9.1         109.8       1.0X
   ```
   
   After:
   
   ```
   OpenJDK 64-Bit Server VM 17.0.16+8-Ubuntu-0ubuntu124.04.1 on Linux 6.14.0-33-generic
   18:39:53.863 ERROR org.apache.spark.util.Utils: Process List(/usr/bin/grep, -m, 1, model name, /proc/cpuinfo) exited with code 1: 
   
   Unknown processor
   LongToUnsafeRowMap metrics:               Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
   ------------------------------------------------------------------------------------------------------------------------
   LongToUnsafeRowMap                                   63            105          38          8.0         125.5       1.0X
   ```
   
   ### 10M Rows
   
   Before: 
   
   ```
   OpenJDK 64-Bit Server VM 17.0.16+8-Ubuntu-0ubuntu124.04.1 on Linux 6.14.0-33-generic
   18:53:43.292 ERROR org.apache.spark.util.Utils: Process List(/usr/bin/grep, -m, 1, model name, /proc/cpuinfo) exited with code 1: 
   
   Unknown processor
   LongToUnsafeRowMap metrics:               Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
   ------------------------------------------------------------------------------------------------------------------------
   LongToUnsafeRowMap                                 2955           3121         235          3.4         295.5       1.0X
   ```
   
   After:
   
   ```
   OpenJDK 64-Bit Server VM 17.0.16+8-Ubuntu-0ubuntu124.04.1 on Linux 6.14.0-33-generic
   18:54:10.447 ERROR org.apache.spark.util.Utils: Process List(/usr/bin/grep, -m, 1, model name, /proc/cpuinfo) exited with code 1: 
   
   Unknown processor
   LongToUnsafeRowMap metrics:               Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
   ------------------------------------------------------------------------------------------------------------------------
   LongToUnsafeRowMap                                 3048           3336         408          3.3         304.8       1.0X
   ```
   
   -------
   
   > e.g., sometimes it appears JIT to be quite smarter than using direct off heap memory
   
   Yes, and also regarding the benchmark results, I do think the new approach has to be at least a bit slower. Do you have any suggestions?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #52817:
URL: https://github.com/apache/spark/pull/52817#issuecomment-3519895892

   Just a note to @zhztheplayer . From my understanding, this feature needs enough time to do extensive tests over various workloads. Given that, I'd like to recommend to re-target this to Apache Spark 4.2.0 only. For 4.2.0, we can get more chances to test this in `4.2.0-preview1`, `4.2.0-preview2`, and so on. For Spark 4.1, it looks too late to me.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #52817:
URL: https://github.com/apache/spark/pull/52817#issuecomment-3574599410

   Thanks! @cloud-fan @yaooqinn @dongjoon-hyun @HyukjinKwon @viirya


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #52817:
URL: https://github.com/apache/spark/pull/52817#issuecomment-3540905799

   The CI failure (hanged forever after changing to on-heap) is being addressed by another PR: https://github.com/apache/spark/pull/53065, which this one depends on.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #52817:
URL: https://github.com/apache/spark/pull/52817#discussion_r2557253807


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala:
##########
@@ -985,6 +975,26 @@ private[execution] final class LongToUnsafeRowMap(
   override def read(kryo: Kryo, in: Input): Unit = {
     read(() => in.readBoolean(), () => in.readLong(), in.readBytes)
   }
+
+  private class UnsafeLongArray(val length: Int) {

Review Comment:
   Updated. Thanks for the suggestion.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #52817:
URL: https://github.com/apache/spark/pull/52817#issuecomment-3513166772

   I wonder if the benchmark can be done at least. I worked on similar changes to implement off heap stuff, and realised that it isn't necessarily fast.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #52817:
URL: https://github.com/apache/spark/pull/52817#issuecomment-3539836782

   > The PR only affects ShuffledHashJoin. In BroadcastHashJoin, the hashed relations are not closed explicitly but are managed by GC. So it will require a different approach to allocate from off-heap.
   
   How is this implemented in this PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #52817:
URL: https://github.com/apache/spark/pull/52817#discussion_r2558129392


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala:
##########
@@ -985,6 +975,26 @@ private[execution] final class LongToUnsafeRowMap(
   override def read(kryo: Kryo, in: Input): Unit = {
     read(() => in.readBoolean(), () => in.readLong(), in.readBytes)
   }
+
+  private class UnsafeLongArray(val length: Int) {

Review Comment:
   It's a few more code changes using `LongArray` - but yes the functinality is the same.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #52817:
URL: https://github.com/apache/spark/pull/52817#discussion_r2555705386


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala:
##########
@@ -985,6 +975,26 @@ private[execution] final class LongToUnsafeRowMap(
   override def read(kryo: Kryo, in: Input): Unit = {
     read(() => in.readBoolean(), () => in.readLong(), in.readBytes)
   }
+
+  private class UnsafeLongArray(val length: Int) {

Review Comment:
   same question. LongArray supports both on-heap and off-heap.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #52817:
URL: https://github.com/apache/spark/pull/52817#issuecomment-3540854361

   Hi @cloud-fan, thanks for having a look.
   
   > How is this implemented in this PR? I don't see any branching code regarding different joins.
   
   This is a bit subtle due to how the task memory manager it got in Spark code.
   
   ## For SHJ
    https://github.com/apache/spark/blob/722bcc0f0d15245a39fae62c0c1c764e4b6a02f8/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/ShuffledHashJoinExec.scala#L108-L116
   
   As seen, `context.taskMemoryManager()` is passed in, and because we [modified](https://github.com/apache/spark/pull/52817/files#diff-127291a0287f790755be5473765ea03eb65f8b58b9ec0760955f124e21e3452fR539) the `LongHashedRelation` to use the tmm's memory mode, so after the PR, SHJ will use off-heap if `spark.memory.offHeap.enabled=false`.
   
   ## For BHJ (Driver)
   
   The code goes through this path: 
   
   https://github.com/apache/spark/blob/722bcc0f0d15245a39fae62c0c1c764e4b6a02f8/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala#L197
   
   then
   
   https://github.com/apache/spark/blob/722bcc0f0d15245a39fae62c0c1c764e4b6a02f8/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala#L1157-L1166
   
   , where no tmm is passed for creating the hashed relation. In this case, a temporary on-heap tmm will be created and used: https://github.com/apache/spark/blob/722bcc0f0d15245a39fae62c0c1c764e4b6a02f8/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala#L142-L150.
   
   ## For BHJ (Executor)
   
   Similar to the driver side, the deserialization code also uses a temporary on-heap tmm: https://github.com/apache/spark/blob/722bcc0f0d15245a39fae62c0c1c764e4b6a02f8/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala#L401-L407.
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52817:
URL: https://github.com/apache/spark/pull/52817#discussion_r2555451772


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala:
##########
@@ -985,6 +975,26 @@ private[execution] final class LongToUnsafeRowMap(
   override def read(kryo: Kryo, in: Input): Unit = {
     read(() => in.readBoolean(), () => in.readLong(), in.readBytes)
   }
+
+  private class UnsafeLongArray(val length: Int) {

Review Comment:
   this is already a `LongArray` class in Spark, can we reuse it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #52817:
URL: https://github.com/apache/spark/pull/52817#issuecomment-3520830787

   > Just a note to @zhztheplayer . From my understanding, this feature needs enough time to do extensive tests over various workloads. Given that, I'd like to recommend to re-target this to Apache Spark 4.2.0 only. For 4.2.0, we can get more chances to test this in `4.2.0-preview1`, `4.2.0-preview2`, and so on. For Spark 4.1, it looks too late to me.
   
   @dongjoon-hyun According to the developer document mentioned in the [issue](https://issues.apache.org/jira/browse/SPARK-54116), I will leave the target version empty. Thank you for helping with planning the target version.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52817:
URL: https://github.com/apache/spark/pull/52817#discussion_r2555451772


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala:
##########
@@ -985,6 +975,26 @@ private[execution] final class LongToUnsafeRowMap(
   override def read(kryo: Kryo, in: Input): Unit = {
     read(() => in.readBoolean(), () => in.readLong(), in.readBytes)
   }
+
+  private class UnsafeLongArray(val length: Int) {

Review Comment:
   there is already a `LongArray` class in Spark, can we reuse it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #52817:
URL: https://github.com/apache/spark/pull/52817#issuecomment-3513149387

   @cloud-fan @yaooqinn @dongjoon-hyun @@HyukjinKwon @viirya
   
   Would you kindly help review this PR? It's for making Spark SQL work more smoothly with 3rd party off-heap based operators / expressions. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation to avoid JVM heap OOMs under off-heap memory mode [spark]

Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on code in PR #52817:
URL: https://github.com/apache/spark/pull/52817#discussion_r2512661397


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala:
##########
@@ -583,24 +584,15 @@ private[execution] final class LongToUnsafeRowMap(
       0)
   }
 
-  private def ensureAcquireMemory(size: Long): Unit = {
-    // do not support spilling
-    val got = acquireMemory(size)
-    if (got < size) {
-      freeMemory(got)
-      throw QueryExecutionErrors.cannotAcquireMemoryToBuildLongHashedRelationError(size, got)

Review Comment:
   We can remove this error class



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org