You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "JkSelf (via GitHub)" <gi...@apache.org> on 2026/04/08 10:53:21 UTC
[PR] [GLUTEN-11888] [VL] Parallel build hash table to improve bhj performance [gluten]
JkSelf opened a new pull request, #11889:
URL: https://github.com/apache/gluten/pull/11889
<!--
Thank you for submitting a pull request! Here are some tips:
1. For first-time contributors, please read our contributing guide:
https://github.com/apache/gluten/blob/main/CONTRIBUTING.md
2. If necessary, create a GitHub issue for discussion beforehand to avoid duplicate work.
3. If the PR is specific to a single backend, include [VL] or [CH] in the PR title to indicate the
Velox or ClickHouse backend, respectively.
4. If the PR is not ready for review, please mark it as a draft.
-->
## What changes are proposed in this pull request?
<!--
Provide a clear and concise description of the changes introduced in this PR.
Ensure the PR description aligns with the code changes, especially after updates.
If applicable, include "Fixes #<GitHub_Issue_ID>" to automatically close the corresponding issue
when the PR is merged.
-->
1. Add buildThreads and build hash table time metrics.
2. Introduce `spark.gluten.velox.broadcast.build.targetBytesPerThrea`configuration to dynamically calculate hash table thread counts based on raw data size
## How was this patch tested?
<!--
Describe how the changes were tested, if applicable.
Include new tests to validate the functionality, if necessary.
For UI-related changes, attach screenshots to demonstrate the updates.
-->
Existing tests
## Was this patch authored or co-authored using generative AI tooling?
No
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11888] [VL] Parallel build hash table to improve bhj performance [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11889:
URL: https://github.com/apache/gluten/pull/11889#issuecomment-4213246783
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11888] [VL] Parallel build hash table to improve bhj performance [gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #11889:
URL: https://github.com/apache/gluten/pull/11889#discussion_r3052618632
##########
backends-velox/src/main/scala/org/apache/gluten/config/VeloxConfig.scala:
##########
@@ -206,12 +206,11 @@ object VeloxConfig extends ConfigRegistry {
.intConf
.createOptional
- val COLUMNAR_VELOX_BROADCAST_HASH_TABLE_BUILD_THREADS =
- buildStaticConf("spark.gluten.sql.columnar.backend.velox.broadcastHashTableBuildThreads")
- .doc("The number of threads used to build the broadcast hash table. " +
- "If not set or set to 0, it will use the default number of threads (available processors).")
- .intConf
- .createWithDefault(1)
+ val COLUMNAR_VELOX_BROADCAST_HASH_TABLE_BUILD_TARGET_BYTES =
+ buildStaticConf("spark.gluten.velox.broadcast.build.targetBytesPerThread")
Review Comment:
Please add it to the config description
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11888] [VL] Parallel build hash table to improve bhj performance [gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #11889:
URL: https://github.com/apache/gluten/pull/11889#discussion_r3052136473
##########
backends-velox/src/main/scala/org/apache/gluten/config/VeloxConfig.scala:
##########
@@ -206,12 +206,11 @@ object VeloxConfig extends ConfigRegistry {
.intConf
.createOptional
- val COLUMNAR_VELOX_BROADCAST_HASH_TABLE_BUILD_THREADS =
- buildStaticConf("spark.gluten.sql.columnar.backend.velox.broadcastHashTableBuildThreads")
- .doc("The number of threads used to build the broadcast hash table. " +
- "If not set or set to 0, it will use the default number of threads (available processors).")
- .intConf
- .createWithDefault(1)
+ val COLUMNAR_VELOX_BROADCAST_HASH_TABLE_BUILD_TARGET_BYTES =
+ buildStaticConf("spark.gluten.velox.broadcast.build.targetBytesPerThread")
Review Comment:
Do we need to adjust the value on some conditions? If yes, please describe how to set the advised value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11888] [VL] Parallel build hash table to improve bhj performance [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11889:
URL: https://github.com/apache/gluten/pull/11889#issuecomment-4206823208
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11888] [VL] Parallel build hash table to improve bhj performance [gluten]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on code in PR #11889:
URL: https://github.com/apache/gluten/pull/11889#discussion_r3052513326
##########
backends-velox/src/main/scala/org/apache/gluten/config/VeloxConfig.scala:
##########
@@ -206,12 +206,11 @@ object VeloxConfig extends ConfigRegistry {
.intConf
.createOptional
- val COLUMNAR_VELOX_BROADCAST_HASH_TABLE_BUILD_THREADS =
- buildStaticConf("spark.gluten.sql.columnar.backend.velox.broadcastHashTableBuildThreads")
- .doc("The number of threads used to build the broadcast hash table. " +
- "If not set or set to 0, it will use the default number of threads (available processors).")
- .intConf
- .createWithDefault(1)
+ val COLUMNAR_VELOX_BROADCAST_HASH_TABLE_BUILD_TARGET_BYTES =
+ buildStaticConf("spark.gluten.velox.broadcast.build.targetBytesPerThread")
Review Comment:
Based on our testing across various thresholds (1MB to 128MB), we recommend a value of 32MB or 64MB, as these consistently provided the most significant performance gains.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11888] [VL] Parallel build hash table to improve bhj performance [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11889:
URL: https://github.com/apache/gluten/pull/11889#issuecomment-4205722514
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11888] [VL] Parallel build hash table to improve bhj performance [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11889:
URL: https://github.com/apache/gluten/pull/11889#issuecomment-4213276312
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11888] [VL] Parallel build hash table to improve bhj performance [gluten]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf merged PR #11889:
URL: https://github.com/apache/gluten/pull/11889
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11888] [VL] Parallel build hash table to improve bhj performance [gluten]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on PR #11889:
URL: https://github.com/apache/gluten/pull/11889#issuecomment-4207539985
@jinchengchenghh Already updated the performance data in PR description. Please help to review again. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org