You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhztheplayer (via GitHub)" <gi...@apache.org> on 2025/10/31 12:24:49 UTC
[PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]
zhztheplayer opened a new pull request, #52817:
URL: https://github.com/apache/spark/pull/52817
### What changes were proposed in this pull request?
The PR adds off-heap memory mode support for LongHashedRelation.
### Why are the changes needed?
To avoid on-heap OOMs when user sets `spark.memory.offHeap.enabled=true`, and configures a comparatively small JVM heap size.
### Does this PR introduce _any_ user-facing change?
When `spark.memory.offHeap.enabled=true` is set:
- Required off-heap memory size may increase
- Required on-heap memory size may decrease
### How was this patch tested?
WIP
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52817:
URL: https://github.com/apache/spark/pull/52817#discussion_r2555447358
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala:
##########
@@ -816,35 +809,34 @@ private[execution] final class LongToUnsafeRowMap(
} else {
// there are some values for this key, put the address in the front of them.
val pointer = toOffset(address) + toSize(address)
- Platform.putLong(page, pointer, array(pos + 1))
+ Platform.putLong(page.getBaseObject, page.getBaseOffset + pointer, array(pos + 1))
array(pos + 1) = address
}
}
private def grow(inputRowSize: Int): Unit = {
// There is 8 bytes for the pointer to next value
- val neededNumWords = (cursor - Platform.LONG_ARRAY_OFFSET + 8 + inputRowSize + 7) / 8
- if (neededNumWords > page.length) {
+ val usedBytes = cursor - page.getBaseOffset
+ val neededNumWords = (usedBytes + 8 + inputRowSize + 7) / 8
+ if (neededNumWords > page.size() / 8) {
if (neededNumWords > (1 << 30)) {
throw QueryExecutionErrors.cannotBuildHashedRelationLargerThan8GError()
}
- val newNumWords = math.max(neededNumWords, math.min(page.length * 2, 1 << 30))
- ensureAcquireMemory(newNumWords * 8L)
- val newPage = new Array[Long](newNumWords.toInt)
- Platform.copyMemory(page, Platform.LONG_ARRAY_OFFSET, newPage, Platform.LONG_ARRAY_OFFSET,
- cursor - Platform.LONG_ARRAY_OFFSET)
- val used = page.length
+ val newNumWords = math.max(neededNumWords, math.min(page.size() / 8 * 2, 1 << 30))
+ val newPage = allocatePage(newNumWords.toInt * 8)
+ Platform.copyMemory(page.getBaseObject, page.getBaseOffset, newPage.getBaseObject,
+ newPage.getBaseOffset, usedBytes)
+ freePage(page)
page = newPage
- freeMemory(used * 8L)
+ cursor = page.getBaseOffset + usedBytes
}
}
private def growArray(): Unit = {
var old_array = array
val n = array.length
numKeys = 0
- ensureAcquireMemory(n * 2 * 8L)
- array = new Array[Long](n * 2)
+ array = new UnsafeLongArray(n * 2)
Review Comment:
Does `UnsafeLongArray` allocate memory from the memory manager?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #52817:
URL: https://github.com/apache/spark/pull/52817#issuecomment-3513284335
> e.g., sometimes it appears JIT to be quite smarter than using direct off heap memory
I actually didn't expect the off-heap relation can be faster - it's more a work to make sure Spark can work with a relatively smaller heap under off-heap memory mode, so we can prevent heap OOMs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #52817: [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation
URL: https://github.com/apache/spark/pull/52817
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #52817:
URL: https://github.com/apache/spark/pull/52817#issuecomment-3513191628
e.g., sometimes it appears JIT to be quite smarter than using direct off heap memory
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #52817:
URL: https://github.com/apache/spark/pull/52817#issuecomment-3573896164
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation to avoid JVM heap OOMs under off-heap memory mode [spark]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #52817:
URL: https://github.com/apache/spark/pull/52817#issuecomment-3517111427
@HyukjinKwon, It's interesting that, after the change, the off-heap mode relation looks meaningfully faster than on-heap relation in the benchmark. I will update the benchmark result inline and in PR description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #52817:
URL: https://github.com/apache/spark/pull/52817#discussion_r2558129392
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala:
##########
@@ -985,6 +975,26 @@ private[execution] final class LongToUnsafeRowMap(
override def read(kryo: Kryo, in: Input): Unit = {
read(() => in.readBoolean(), () => in.readLong(), in.readBytes)
}
+
+ private class UnsafeLongArray(val length: Int) {
Review Comment:
It's a bit more code changes using `LongArray` - but yes the functinality is the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #52817:
URL: https://github.com/apache/spark/pull/52817#discussion_r2557248391
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala:
##########
@@ -816,35 +809,34 @@ private[execution] final class LongToUnsafeRowMap(
} else {
// there are some values for this key, put the address in the front of them.
val pointer = toOffset(address) + toSize(address)
- Platform.putLong(page, pointer, array(pos + 1))
+ Platform.putLong(page.getBaseObject, page.getBaseOffset + pointer, array(pos + 1))
array(pos + 1) = address
}
}
private def grow(inputRowSize: Int): Unit = {
// There is 8 bytes for the pointer to next value
- val neededNumWords = (cursor - Platform.LONG_ARRAY_OFFSET + 8 + inputRowSize + 7) / 8
- if (neededNumWords > page.length) {
+ val usedBytes = cursor - page.getBaseOffset
+ val neededNumWords = (usedBytes + 8 + inputRowSize + 7) / 8
+ if (neededNumWords > page.size() / 8) {
if (neededNumWords > (1 << 30)) {
throw QueryExecutionErrors.cannotBuildHashedRelationLargerThan8GError()
}
- val newNumWords = math.max(neededNumWords, math.min(page.length * 2, 1 << 30))
- ensureAcquireMemory(newNumWords * 8L)
- val newPage = new Array[Long](newNumWords.toInt)
- Platform.copyMemory(page, Platform.LONG_ARRAY_OFFSET, newPage, Platform.LONG_ARRAY_OFFSET,
- cursor - Platform.LONG_ARRAY_OFFSET)
- val used = page.length
+ val newNumWords = math.max(neededNumWords, math.min(page.size() / 8 * 2, 1 << 30))
+ val newPage = allocatePage(newNumWords.toInt * 8)
+ Platform.copyMemory(page.getBaseObject, page.getBaseOffset, newPage.getBaseObject,
+ newPage.getBaseOffset, usedBytes)
+ freePage(page)
page = newPage
- freeMemory(used * 8L)
+ cursor = page.getBaseOffset + usedBytes
}
}
private def growArray(): Unit = {
var old_array = array
val n = array.length
numKeys = 0
- ensureAcquireMemory(n * 2 * 8L)
- array = new Array[Long](n * 2)
+ array = new UnsafeLongArray(n * 2)
Review Comment:
Yes, it calls the memory manager in its constructor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #52817:
URL: https://github.com/apache/spark/pull/52817#issuecomment-3513268573
@HyukjinKwon Thank you for the quick response.
I benchmarked using the existing HashedRelationMetricsBenchmark.
### 500K Rows
Before:
```
OpenJDK 64-Bit Server VM 17.0.16+8-Ubuntu-0ubuntu124.04.1 on Linux 6.14.0-33-generic
18:40:31.460 ERROR org.apache.spark.util.Utils: Process List(/usr/bin/grep, -m, 1, model name, /proc/cpuinfo) exited with code 1:
Unknown processor
LongToUnsafeRowMap metrics: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
LongToUnsafeRowMap 55 63 4 9.1 109.8 1.0X
```
After:
```
OpenJDK 64-Bit Server VM 17.0.16+8-Ubuntu-0ubuntu124.04.1 on Linux 6.14.0-33-generic
18:39:53.863 ERROR org.apache.spark.util.Utils: Process List(/usr/bin/grep, -m, 1, model name, /proc/cpuinfo) exited with code 1:
Unknown processor
LongToUnsafeRowMap metrics: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
LongToUnsafeRowMap 63 105 38 8.0 125.5 1.0X
```
### 10M Rows
Before:
```
OpenJDK 64-Bit Server VM 17.0.16+8-Ubuntu-0ubuntu124.04.1 on Linux 6.14.0-33-generic
18:53:43.292 ERROR org.apache.spark.util.Utils: Process List(/usr/bin/grep, -m, 1, model name, /proc/cpuinfo) exited with code 1:
Unknown processor
LongToUnsafeRowMap metrics: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
LongToUnsafeRowMap 2955 3121 235 3.4 295.5 1.0X
```
After:
```
OpenJDK 64-Bit Server VM 17.0.16+8-Ubuntu-0ubuntu124.04.1 on Linux 6.14.0-33-generic
18:54:10.447 ERROR org.apache.spark.util.Utils: Process List(/usr/bin/grep, -m, 1, model name, /proc/cpuinfo) exited with code 1:
Unknown processor
LongToUnsafeRowMap metrics: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
LongToUnsafeRowMap 3048 3336 408 3.3 304.8 1.0X
```
-------
> e.g., sometimes it appears JIT to be quite smarter than using direct off heap memory
Yes, and also regarding the benchmark results, I do think the new approach has to be at least a bit slower. Do you have any suggestions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #52817:
URL: https://github.com/apache/spark/pull/52817#issuecomment-3519895892
Just a note to @zhztheplayer . From my understanding, this feature needs enough time to do extensive tests over various workloads. Given that, I'd like to recommend to re-target this to Apache Spark 4.2.0 only. For 4.2.0, we can get more chances to test this in `4.2.0-preview1`, `4.2.0-preview2`, and so on. For Spark 4.1, it looks too late to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #52817:
URL: https://github.com/apache/spark/pull/52817#issuecomment-3574599410
Thanks! @cloud-fan @yaooqinn @dongjoon-hyun @HyukjinKwon @viirya
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #52817:
URL: https://github.com/apache/spark/pull/52817#issuecomment-3540905799
The CI failure (hanged forever after changing to on-heap) is being addressed by another PR: https://github.com/apache/spark/pull/53065, which this one depends on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #52817:
URL: https://github.com/apache/spark/pull/52817#discussion_r2557253807
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala:
##########
@@ -985,6 +975,26 @@ private[execution] final class LongToUnsafeRowMap(
override def read(kryo: Kryo, in: Input): Unit = {
read(() => in.readBoolean(), () => in.readLong(), in.readBytes)
}
+
+ private class UnsafeLongArray(val length: Int) {
Review Comment:
Updated. Thanks for the suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #52817:
URL: https://github.com/apache/spark/pull/52817#issuecomment-3513166772
I wonder if the benchmark can be done at least. I worked on similar changes to implement off heap stuff, and realised that it isn't necessarily fast.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #52817:
URL: https://github.com/apache/spark/pull/52817#issuecomment-3539836782
> The PR only affects ShuffledHashJoin. In BroadcastHashJoin, the hashed relations are not closed explicitly but are managed by GC. So it will require a different approach to allocate from off-heap.
How is this implemented in this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #52817:
URL: https://github.com/apache/spark/pull/52817#discussion_r2558129392
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala:
##########
@@ -985,6 +975,26 @@ private[execution] final class LongToUnsafeRowMap(
override def read(kryo: Kryo, in: Input): Unit = {
read(() => in.readBoolean(), () => in.readLong(), in.readBytes)
}
+
+ private class UnsafeLongArray(val length: Int) {
Review Comment:
It's a few more code changes using `LongArray` - but yes the functinality is the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #52817:
URL: https://github.com/apache/spark/pull/52817#discussion_r2555705386
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala:
##########
@@ -985,6 +975,26 @@ private[execution] final class LongToUnsafeRowMap(
override def read(kryo: Kryo, in: Input): Unit = {
read(() => in.readBoolean(), () => in.readLong(), in.readBytes)
}
+
+ private class UnsafeLongArray(val length: Int) {
Review Comment:
same question. LongArray supports both on-heap and off-heap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #52817:
URL: https://github.com/apache/spark/pull/52817#issuecomment-3540854361
Hi @cloud-fan, thanks for having a look.
> How is this implemented in this PR? I don't see any branching code regarding different joins.
This is a bit subtle due to how the task memory manager it got in Spark code.
## For SHJ
https://github.com/apache/spark/blob/722bcc0f0d15245a39fae62c0c1c764e4b6a02f8/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/ShuffledHashJoinExec.scala#L108-L116
As seen, `context.taskMemoryManager()` is passed in, and because we [modified](https://github.com/apache/spark/pull/52817/files#diff-127291a0287f790755be5473765ea03eb65f8b58b9ec0760955f124e21e3452fR539) the `LongHashedRelation` to use the tmm's memory mode, so after the PR, SHJ will use off-heap if `spark.memory.offHeap.enabled=false`.
## For BHJ (Driver)
The code goes through this path:
https://github.com/apache/spark/blob/722bcc0f0d15245a39fae62c0c1c764e4b6a02f8/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala#L197
then
https://github.com/apache/spark/blob/722bcc0f0d15245a39fae62c0c1c764e4b6a02f8/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala#L1157-L1166
, where no tmm is passed for creating the hashed relation. In this case, a temporary on-heap tmm will be created and used: https://github.com/apache/spark/blob/722bcc0f0d15245a39fae62c0c1c764e4b6a02f8/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala#L142-L150.
## For BHJ (Executor)
Similar to the driver side, the deserialization code also uses a temporary on-heap tmm: https://github.com/apache/spark/blob/722bcc0f0d15245a39fae62c0c1c764e4b6a02f8/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala#L401-L407.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52817:
URL: https://github.com/apache/spark/pull/52817#discussion_r2555451772
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala:
##########
@@ -985,6 +975,26 @@ private[execution] final class LongToUnsafeRowMap(
override def read(kryo: Kryo, in: Input): Unit = {
read(() => in.readBoolean(), () => in.readLong(), in.readBytes)
}
+
+ private class UnsafeLongArray(val length: Int) {
Review Comment:
this is already a `LongArray` class in Spark, can we reuse it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #52817:
URL: https://github.com/apache/spark/pull/52817#issuecomment-3520830787
> Just a note to @zhztheplayer . From my understanding, this feature needs enough time to do extensive tests over various workloads. Given that, I'd like to recommend to re-target this to Apache Spark 4.2.0 only. For 4.2.0, we can get more chances to test this in `4.2.0-preview1`, `4.2.0-preview2`, and so on. For Spark 4.1, it looks too late to me.
@dongjoon-hyun According to the developer document mentioned in the [issue](https://issues.apache.org/jira/browse/SPARK-54116), I will leave the target version empty. Thank you for helping with planning the target version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52817:
URL: https://github.com/apache/spark/pull/52817#discussion_r2555451772
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala:
##########
@@ -985,6 +975,26 @@ private[execution] final class LongToUnsafeRowMap(
override def read(kryo: Kryo, in: Input): Unit = {
read(() => in.readBoolean(), () => in.readLong(), in.readBytes)
}
+
+ private class UnsafeLongArray(val length: Int) {
Review Comment:
there is already a `LongArray` class in Spark, can we reuse it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation [spark]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #52817:
URL: https://github.com/apache/spark/pull/52817#issuecomment-3513149387
@cloud-fan @yaooqinn @dongjoon-hyun @@HyukjinKwon @viirya
Would you kindly help review this PR? It's for making Spark SQL work more smoothly with 3rd party off-heap based operators / expressions. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation to avoid JVM heap OOMs under off-heap memory mode [spark]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on code in PR #52817:
URL: https://github.com/apache/spark/pull/52817#discussion_r2512661397
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala:
##########
@@ -583,24 +584,15 @@ private[execution] final class LongToUnsafeRowMap(
0)
}
- private def ensureAcquireMemory(size: Long): Unit = {
- // do not support spilling
- val got = acquireMemory(size)
- if (got < size) {
- freeMemory(got)
- throw QueryExecutionErrors.cannotAcquireMemoryToBuildLongHashedRelationError(size, got)
Review Comment:
We can remove this error class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org