You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "JkSelf (via GitHub)" <gi...@apache.org> on 2026/08/13 15:42:01 UTC
[PR] [VL]Optimize driver side build hash table performance [gluten]
JkSelf opened a new pull request, #12765:
URL: https://github.com/apache/gluten/pull/12765
<!--
Thank you for submitting a pull request! Here are some tips:
1. For first-time contributors, please read our contributing guide:
https://github.com/apache/gluten/blob/main/CONTRIBUTING.md
2. If necessary, create a GitHub issue for discussion beforehand to avoid duplicate work.
3. If the PR is specific to a single backend, include [VL] or [CH] in the PR title to indicate the
Velox or ClickHouse backend, respectively.
4. If the PR is not ready for review, please mark it as a draft.
-->
## What changes are proposed in this pull request?
<!--
Provide a clear and concise description of the changes introduced in this PR.
Ensure the PR description aligns with the code changes, especially after updates.
If applicable, include "Fixes #<GitHub_Issue_ID>" to automatically close the corresponding issue
when the PR is merged.
-->
## How was this patch tested?
<!--
Describe how the changes were tested, if applicable.
Include new tests to validate the functionality, if necessary.
For UI-related changes, attach screenshots to demonstrate the updates.
-->
## Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL]Optimize driver side build hash table performance [gluten]
Posted by "github-actions[bot] (via GitHub) via commits" <co...@gluten.apache.org>.
github-actions[bot] closed pull request #12765: [VL]Optimize driver side build hash table performance
URL: https://github.com/apache/gluten/pull/12765
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL]Optimize driver side build hash table performance [gluten]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12765:
URL: https://github.com/apache/gluten/pull/12765#discussion_r3902900869
##########
backends-velox/src/main/scala/org/apache/gluten/execution/VeloxBroadcastBuildSideCache.scala:
##########
@@ -212,6 +250,50 @@ object VeloxBroadcastBuildSideCache
)
}
+ /**
+ * Returns a handle to the native hash table for `serialized`, deserializing it at most once per
+ * driver-side broadcast id. The returned handle is owned by [[sharedDeserializedCache]]; callers
+ * must clone it rather than releasing it.
+ */
+ private def getOrDeserializeShared(
+ serialized: SerializedBroadcastHashTable,
+ broadcastHashTableId: String,
+ deserializeHashTableTimeMetric: Option[org.apache.spark.sql.execution.metric.SQLMetric])
+ : SharedDeserializedHashTable = {
+ // Older payloads, and any path that did not go through the driver-side build, carry no
+ // broadcast id. Fall back to keying on the join id, which is what the previous behavior was.
+ val sharedKey =
+ if (serialized.broadcastId != null) serialized.broadcastId else broadcastHashTableId
+
+ sharedDeserializedCache.get(
+ sharedKey,
+ (key: String) => {
+ logInfo(s"Deserializing hash table on executor for broadcast ID: $key")
+ val startTime = System.currentTimeMillis()
+ val hashTableHandle = serialized.deserialize(key)
+ val timeMs = System.currentTimeMillis() - startTime
+ deserializeHashTableTimeMetric.foreach(_ += timeMs)
+
+ // The serialized bytes have been fully consumed into the native table. On an executor
+ // nothing else refers to them, so free the off-heap copy instead of waiting for the
+ // broadcast object to be collected. On the driver the same object is still owned by the
+ // broadcast variable and by driverSerializedCache, so it must be left alone.
+ if (!isDriver) {
+ serialized.releaseSerializedData()
+ }
Review Comment:
Releasing `serialized`’s off-heap bytes mutates the broadcast value on executors and can make later re-materialization impossible if the shared native table is evicted (e.g., `expireAfterAccess`) or if Spark reuses the same cached broadcast value for a later stage without forcing a re-deserialize from blocks. This can surface as runtime failures in `SerializedBroadcastHashTable.deserialize` (“already been released”). Prefer keeping `serializedData` intact (remove `releaseSerializedData()`), or only releasing bytes when you can guarantee the native table will remain available for the broadcast’s full usable lifetime (e.g., tie it to broadcast lifecycle/unpersist rather than cache eviction).
##########
gluten-arrow/src/main/java/org/apache/spark/sql/execution/unsafe/UnsafeByteArray.java:
##########
@@ -102,12 +120,12 @@ public void read(Kryo kryo, Input input) {
this.buffer = ArrowBufferAllocators.globalInstance().buffer((int) size);
Review Comment:
Casting `size` (a `long`) to `int` can truncate for payloads > 2GB, leading to an under-allocated `ArrowBuf` and subsequent out-of-bounds reads/writes during streaming. Since the serialized format persists `size` as a `long`, add an explicit bound check (e.g., fail fast with an exception) before allocating and before using the `int index` loop variable. The same issue applies to the identical allocation in `readExternal`.
##########
gluten-arrow/src/main/java/org/apache/spark/sql/execution/unsafe/UnsafeByteArray.java:
##########
@@ -45,9 +49,14 @@ public class UnsafeByteArray implements Externalizable, KryoSerializable {
public UnsafeByteArray() {}
- private byte[] chunkBuf() {
- if (chunkBuf == null) {
- chunkBuf = new byte[CHUNK_SIZE];
+ /**
+ * Returns a scratch buffer of at most {@link #CHUNK_SIZE} bytes, never larger than the payload
+ * itself so that relations holding many small batches do not each allocate a full-sized chunk.
+ */
+ private byte[] chunkBuf(long dataSize) {
+ final int wanted = (int) Math.max(1, Math.min(CHUNK_SIZE, dataSize));
+ if (chunkBuf == null || chunkBuf.length < wanted) {
+ chunkBuf = new byte[wanted];
Review Comment:
The doc comment says the scratch buffer is “never larger than the payload itself”, but the implementation only grows the buffer and never shrinks it; after handling a large payload once, later smaller payloads can still reuse a larger-than-payload `chunkBuf`. Either adjust the wording to reflect the non-shrinking behavior (recommended to avoid realloc churn), or implement shrinking with care (tradeoff: more allocations/GC).
##########
backends-velox/src/main/scala/org/apache/gluten/execution/SerializedBroadcastHashTable.scala:
##########
@@ -82,8 +105,20 @@ class SerializedBroadcastHashTable(
joinHasNullKeys)
}
+ /**
+ * Frees the off-heap buffer holding the serialized bytes. Only safe once the native hash table
+ * has been materialized from it, and only on an executor: on the driver the very same object is
+ * still owned by the broadcast variable and by
+ * [[VeloxBroadcastBuildSideCache.buildAndSerializeOnDriverInBroadcastExchange]]'s cache.
+ */
+ def releaseSerializedData(): Unit = {
+ if (serializedData != null) {
+ serializedData.release()
+ }
+ }
+
/** Get the size of serialized data in bytes. */
- def sizeInBytes: Long = serializedData.size()
+ def sizeInBytes: Long = if (serializedData == null) 0L else serializedData.size()
Review Comment:
After `releaseSerializedData()` runs, `serializedData` can be non-null but released; `sizeInBytes` will still report the original size, which can be misleading for memory accounting/debugging. Consider returning 0 when `serializedData.isReleased` is true (or documenting that `sizeInBytes` reflects the original payload size, not current resident bytes).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL]Optimize driver side build hash table performance [gluten]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12765:
URL: https://github.com/apache/gluten/pull/12765#discussion_r3925848320
##########
backends-velox/src/main/scala/org/apache/gluten/execution/SerializedBroadcastHashTable.scala:
##########
@@ -82,8 +105,20 @@ class SerializedBroadcastHashTable(
joinHasNullKeys)
}
+ /**
+ * Frees the off-heap buffer holding the serialized bytes. Only safe once the native hash table
+ * has been materialized from it, and only on an executor: on the driver the very same object is
+ * still owned by the broadcast variable and by
+ * [[VeloxBroadcastBuildSideCache.buildAndSerializeOnDriverInBroadcastExchange]]'s cache.
+ */
+ def releaseSerializedData(): Unit = {
+ if (serializedData != null) {
+ serializedData.release()
+ }
+ }
+
/** Get the size of serialized data in bytes. */
- def sizeInBytes: Long = serializedData.size()
+ def sizeInBytes: Long = if (serializedData == null) 0L else serializedData.size()
Review Comment:
`releaseSerializedData()` frees the underlying off-heap buffer, but `sizeInBytes` will still report the pre-release size because `UnsafeByteArray.release()` nulls the buffer without changing the `size` field. This can lead to incorrect memory accounting/metrics (and can be misleading in logs/diagnostics). Consider returning 0 when `serializedData.isReleased` is true, or updating `UnsafeByteArray.release()` to also reset `size` when releasing.
##########
cpp/velox/jni/VeloxJniWrapper.cc:
##########
@@ -1208,7 +1220,11 @@ JNIEXPORT jlong JNICALL Java_org_apache_gluten_vectorized_HashJoinBuilder_deseri
reinterpret_cast<const uint8_t*>(address),
static_cast<size_t>(size),
static_cast<bool>(ignoreNullKeys),
- static_cast<bool>(joinHasNullKeys));
+ static_cast<bool>(joinHasNullKeys),
+ // Deserializing on one thread costs about as much as building the table from the raw build
+ // side does on all of them, which would leave the driver-side build with no upside at all.
+ // FIXME: This reuses the io executor which is supposed to only serve async IO tasks.
+ VeloxBackend::get()->ioExecutor());
Review Comment:
This explicitly schedules CPU-heavy deserialization work onto the IO executor (and notes it as a FIXME). That can starve genuine async IO and cause query-wide latency regressions under load. Consider introducing (or using, if already available) a dedicated CPU executor/thread-pool for hash table build/deserialize work, or plumb an appropriate executor from the caller/config instead of reusing `ioExecutor()`.
##########
backends-velox/src/main/scala/org/apache/gluten/execution/VeloxBroadcastBuildSideCache.scala:
##########
@@ -199,11 +234,14 @@ object VeloxBroadcastBuildSideCache
buildSideRelationCache.get(
broadcastHashTableId,
(_: String) => {
- logInfo(s"Deserializing hash table on executor for broadcast ID: $broadcastHashTableId")
- val startTime = System.currentTimeMillis()
- val hashTableHandle = serialized.deserialize(broadcastHashTableId)
- val timeMs = System.currentTimeMillis() - startTime
- deserializeHashTableTimeMetric.foreach(_ += timeMs)
+ val shared = getOrDeserializeShared(
+ serialized,
+ broadcastHashTableId,
+ deserializeHashTableTimeMetric)
+ // Register the shared table under this join's id as well, so that the native probe side
+ // can resolve it. The clone holds its own reference to the same native table.
+ val hashTableHandle =
+ HashJoinBuilder.cloneHashTable(broadcastHashTableId, shared.pointer)
Review Comment:
The new shared-deserialization + per-join clone behavior is performance-critical and subtly interacts with lifecycle (eviction + native ref-counting + `releaseSerializedData()`). Consider adding an integration test that forces a reused broadcast exchange with multiple BHJs on the same executor, and asserts that deserialization happens once per broadcast (e.g., via metrics/log counters) and that subsequent joins use clones successfully even after the serialized bytes are released.
##########
backends-velox/src/main/scala/org/apache/spark/sql/execution/SerializedHashTableBroadcastRelation.scala:
##########
@@ -83,9 +93,15 @@ case class SerializedHashTableBroadcastRelation(
/**
* Transform is used for DPP (Dynamic Partition Pruning) to extract keys. We delegate to the
* underlying buildSideRelation in the serialized hash table.
+ *
+ * Driver-only, for the same reason as [[deserialized]].
*/
override def transform(key: Expression): Array[InternalRow] = {
- serializedHashTable.buildSideRelation.transform(key)
+ val relation = serializedHashTable.buildSideRelation
+ if (relation == null) {
+ throw new IllegalStateException(driverOnlyMessage("transform"))
+ }
+ relation.transform(key)
Review Comment:
After making `buildSideRelation` transient/not serialized, this relation will have `buildSideRelation == null` on executors. Throwing from `transform` (and similarly from `deserialized`) is a behavioral break for any Spark paths that legitimately call these methods on executors (e.g., DPP key extraction in some runtimes). If executors may need these, consider preserving the minimal data required for `transform`/`deserialized` in the broadcast payload (instead of the full raw relation), or implement these methods using the serialized bytes (or explicitly ensure via call sites that they are never invoked off-driver).
##########
cpp/velox/jni/VeloxJniWrapper.cc:
##########
@@ -1297,9 +1313,10 @@ JNIEXPORT void JNICALL Java_org_apache_gluten_vectorized_HashJoinBuilder_seriali
auto builder = ObjectStore::retrieve<gluten::HashTableBuilder>(hashTableHandle);
VELOX_CHECK_GT(address, 0, "Serialized hash table buffer address must be positive");
VELOX_CHECK_GE(size, 0, "Serialized hash table buffer size must be non-negative");
- const auto serializedSize = gluten::serializedHashTableSize(builder);
- VELOX_CHECK_EQ(static_cast<size_t>(size), serializedSize, "Hash table buffer size mismatch");
- gluten::serializeHashTableTo(builder, reinterpret_cast<uint8_t*>(address), serializedSize);
+ // Do not recompute serializedHashTableSize() here just to validate 'size': that walks every
+ // build row again to re-measure the variable-width columns. serializeTo() bounds-checks each
+ // write and asserts that it filled the buffer exactly, which covers the same mistake.
+ gluten::serializeHashTableTo(builder, reinterpret_cast<uint8_t*>(address), static_cast<size_t>(size));
Review Comment:
Removing the explicit size equality check changes error reporting and relies on internal checks that may be asserts (often compiled out in release builds), potentially turning a clear size-mismatch into a harder-to-diagnose failure. If recomputing the size is too expensive, consider caching the computed serialized size at allocation time (and passing it through), or ensure `serializeTo()` uses non-assert runtime checks for both overflow and exact-size agreement so mismatches fail deterministically with a clear message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL]Optimize driver side build hash table performance [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12765:
URL: https://github.com/apache/gluten/pull/12765#issuecomment-6281326803
This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL]Optimize driver side build hash table performance [gluten]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12765:
URL: https://github.com/apache/gluten/pull/12765#discussion_r3776883259
##########
backends-velox/src/main/scala/org/apache/gluten/execution/VeloxBroadcastBuildSideCache.scala:
##########
@@ -189,6 +190,22 @@ object VeloxBroadcastBuildSideCache
}
}
+ /**
+ * Recovers the raw build side relation that belongs to a driver-side built hash table.
+ *
+ * `driverSerializedCache` pins the relation on the driver for the lifetime of the broadcast, so
+ * this returns a non-null value there even if the broadcast block itself had to be re-read from
+ * disk. On executors the cache is always empty and the result is null, which is fine because the
+ * raw batches are never consumed there.
+ */
Review Comment:
This Scaladoc says `driverSerializedCache` "pins the relation on the driver for the lifetime of the broadcast", but the cache is configured with `expireAfterAccess`, so entries can still be evicted/expired while the broadcast is alive. If that happens, `SerializedBroadcastHashTable.readExternal` cannot recover `buildSideRelation` on the driver after a disk reload, and driver-side consumers will see null and fail later.
##########
backends-velox/src/main/scala/org/apache/gluten/execution/SerializedBroadcastHashTable.scala:
##########
@@ -48,8 +62,12 @@ class SerializedBroadcastHashTable(
out.writeBoolean(droppedDuplicates)
out.writeLong(bloomFilterBlocksByteSize)
out.writeLong(hashProbeDynamicFiltersProduced)
+ out.writeUTF(if (driverRelationId == null) "" else driverRelationId)
serializedData.writeExternal(out)
- out.writeObject(buildSideRelation)
+ // `buildSideRelation` is intentionally not written. It holds the raw build side batches, which
+ // are only consumed on the driver (DPP key extraction through `transform`, and fallback to
+ // vanilla Spark through `deserialized`). Executors read the serialized hash table and never
+ // touch the raw batches, so shipping both would roughly double the broadcast payload.
}
Review Comment:
The broadcast payload format and semantics changed (no longer serializing `buildSideRelation`, plus `driverRelationId` recovery and per-JVM handle memoization with `cloneHashTable`/`releaseIfOwner`). There are existing driver-side broadcast hash table tests (e.g. `VeloxHashJoinSuite`), but none that exercise a serialize/deserialize round-trip and validate: (1) executors see `buildSideRelation == null`, (2) the driver can recover `buildSideRelation` after re-read, and (3) multiple `deserialize(cacheKey)` calls clone rather than re-deserialize. Adding a focused unit test would help prevent regressions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL]Optimize driver side build hash table performance [gluten]
Posted by "github-actions[bot] (via GitHub) via commits" <co...@gluten.apache.org>.
github-actions[bot] commented on PR #12765:
URL: https://github.com/apache/gluten/pull/12765#issuecomment-6537197406
This PR was auto-closed because it has been stalled for 10 days with no activity. Please feel free to reopen if it is still valid. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org