You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "sabbasani (via GitHub)" <gi...@apache.org> on 2026/10/14 17:56:36 UTC
[PR] [GLUTEN-12861] Fix GlutenCachedTableSuite silently losing spark.sql.s… [gluten]
sabbasani opened a new pull request, #13287:
URL: https://github.com/apache/gluten/pull/13287
…huffle.partitions=5
sparkConf is a def that creates a new SparkConf on every call. The original override called super.sparkConf twice, setting each property on a different object and discarding the first, so spark.sql.shuffle.partitions=5 was silently lost and fell back to the default of 1.
Fix: call super.sparkConf once and chain both .set() calls on the same object. Also remove the sys.props.put() JVM-global side effect (redundant now that sparkConf correctly propagates the setting) and its misleading comment that claimed to disable the cache when it actually enabled it.
Affects: spark34, spark35, spark40, spark41
<!--
Thank you for submitting a pull request! Here are some tips:
1. For first-time contributors, please read our contributing guide:
https://github.com/apache/gluten/blob/main/CONTRIBUTING.md
2. If necessary, create a GitHub issue for discussion beforehand to avoid duplicate work.
3. If the PR is specific to a single backend, include [VL] or [CH] in the PR title to indicate the
Velox or ClickHouse backend, respectively.
4. If the PR is not ready for review, please mark it as a draft.
-->
## What changes are proposed in this pull request?
<!--
Provide a clear and concise description of the changes introduced in this PR.
Ensure the PR description aligns with the code changes, especially after updates.
If applicable, include "Fixes #<GitHub_Issue_ID>" to automatically close the corresponding issue
when the PR is merged.
-->
## How was this patch tested?
<!--
Describe how the changes were tested, if applicable.
Include new tests to validate the functionality, if necessary.
For UI-related changes, attach screenshots to demonstrate the updates.
-->
## Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12861] Fix GlutenCachedTableSuite silently losing spark.sql.shuffle.partitions=5 [gluten]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #13287:
URL: https://github.com/apache/gluten/pull/13287#discussion_r4397162617
##########
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/GlutenCachedTableSuite.scala:
##########
@@ -31,12 +31,10 @@ class GlutenCachedTableSuite
with GlutenSQLTestsTrait
with AdaptiveSparkPlanHelper {
import testImplicits._
- // for temporarily disable the columnar table cache globally.
- sys.props.put(GlutenConfig.COLUMNAR_TABLE_CACHE_ENABLED.key, "true")
- override def sparkConf: SparkConf = {
- super.sparkConf.set("spark.sql.shuffle.partitions", "5")
- super.sparkConf.set(GlutenConfig.COLUMNAR_TABLE_CACHE_ENABLED.key, "true")
- }
+ override def sparkConf: SparkConf =
+ super.sparkConf
+ .set("spark.sql.shuffle.partitions", "5")
Review Comment:
The spark42 copy of `GlutenCachedTableSuite` still has the old code. `gluten-ut/spark42` landed on main after this branch was cut and only came in through a later merge, so lines 35-38 there still do `sys.props.put` and two separate `super.sparkConf.set(...)` calls. The 4.2 `VeloxTestSettings` enables this suite, so after merge it keeps running with 1 shuffle partition on Spark 4.2, and #12861 would be closed with that copy still broken.
Could you apply the same change to spark42 and add spark42 to the Affects list in the description? `spark-test-spark42` is skipped on this PR because none of the changed paths trigger it; touching `gluten-ut/spark42/` will.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12861] Fix GlutenCachedTableSuite silently losing spark.sql.s… [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13287:
URL: https://github.com/apache/gluten/pull/13287#issuecomment-6411963637
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12861] Fix GlutenCachedTableSuite silently losing spark.sql.shuffle.partitions=5 [gluten]
Posted by "sabbasani (via GitHub)" <gi...@apache.org>.
sabbasani commented on code in PR #13287:
URL: https://github.com/apache/gluten/pull/13287#discussion_r4397538475
##########
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/GlutenCachedTableSuite.scala:
##########
@@ -31,12 +31,10 @@ class GlutenCachedTableSuite
with GlutenSQLTestsTrait
with AdaptiveSparkPlanHelper {
import testImplicits._
- // for temporarily disable the columnar table cache globally.
- sys.props.put(GlutenConfig.COLUMNAR_TABLE_CACHE_ENABLED.key, "true")
- override def sparkConf: SparkConf = {
- super.sparkConf.set("spark.sql.shuffle.partitions", "5")
- super.sparkConf.set(GlutenConfig.COLUMNAR_TABLE_CACHE_ENABLED.key, "true")
- }
+ override def sparkConf: SparkConf =
+ super.sparkConf
+ .set("spark.sql.shuffle.partitions", "5")
Review Comment:
@LuciferYang Done — applied the same fix to gluten-ut/spark42/src/test/scala/org/apache/spark/sql/GlutenCachedTableSuite.scala and updated the Affects: line in the commit message to include spark42. The file now uses the chained super.sparkConf form with no sys.props.put, matching the spark34/35/40/41 copies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12861] [TEST] Fix GlutenCachedTableSuite silently losing spark.sql.shuffle.partitions=5 [gluten]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #13287:
URL: https://github.com/apache/gluten/pull/13287#issuecomment-6515975425
Merged into main. Thanks @sabbasani
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12861] Fix GlutenCachedTableSuite silently losing spark.sql.shuffle.partitions=5 [gluten]
Posted by "sabbasani (via GitHub)" <gi...@apache.org>.
sabbasani commented on PR #13287:
URL: https://github.com/apache/gluten/pull/13287#issuecomment-6515880994
> @sabbasani Could you update the PR description to follow the template format?
@LuciferYang I have updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12861] Fix GlutenCachedTableSuite silently losing spark.sql.shuffle.partitions=5 [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13287:
URL: https://github.com/apache/gluten/pull/13287#issuecomment-6462110157
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12861] Fix GlutenCachedTableSuite silently losing spark.sql.shuffle.partitions=5 [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13287:
URL: https://github.com/apache/gluten/pull/13287#issuecomment-6446917793
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12861] Fix GlutenCachedTableSuite silently losing spark.sql.shuffle.partitions=5 [gluten]
Posted by "sabbasani (via GitHub)" <gi...@apache.org>.
sabbasani commented on PR #13287:
URL: https://github.com/apache/gluten/pull/13287#issuecomment-6460935500
@LuciferYang can you review PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12861] Fix GlutenCachedTableSuite silently losing spark.sql.shuffle.partitions=5 [gluten]
Posted by "sabbasani (via GitHub)" <gi...@apache.org>.
sabbasani commented on code in PR #13287:
URL: https://github.com/apache/gluten/pull/13287#discussion_r4397538475
##########
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/GlutenCachedTableSuite.scala:
##########
@@ -31,12 +31,10 @@ class GlutenCachedTableSuite
with GlutenSQLTestsTrait
with AdaptiveSparkPlanHelper {
import testImplicits._
- // for temporarily disable the columnar table cache globally.
- sys.props.put(GlutenConfig.COLUMNAR_TABLE_CACHE_ENABLED.key, "true")
- override def sparkConf: SparkConf = {
- super.sparkConf.set("spark.sql.shuffle.partitions", "5")
- super.sparkConf.set(GlutenConfig.COLUMNAR_TABLE_CACHE_ENABLED.key, "true")
- }
+ override def sparkConf: SparkConf =
+ super.sparkConf
+ .set("spark.sql.shuffle.partitions", "5")
Review Comment:
Done — applied the same fix to gluten-ut/spark42/src/test/scala/org/apache/spark/sql/GlutenCachedTableSuite.scala and updated the Affects: line in the commit message to include spark42. The file now uses the chained super.sparkConf form with no sys.props.put, matching the spark34/35/40/41 copies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12861] Fix GlutenCachedTableSuite silently losing spark.sql.s… [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13287:
URL: https://github.com/apache/gluten/pull/13287#issuecomment-6186594732
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12861] Fix GlutenCachedTableSuite silently losing spark.sql.shuffle.partitions=5 [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13287:
URL: https://github.com/apache/gluten/pull/13287#issuecomment-6469832866
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12861] Fix GlutenCachedTableSuite silently losing spark.sql.shuffle.partitions=5 [gluten]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #13287:
URL: https://github.com/apache/gluten/pull/13287#issuecomment-6514757617
@sabbasani Could you update the PR description to follow the template format?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12861] Fix GlutenCachedTableSuite silently losing spark.sql.shuffle.partitions=5 [gluten]
Posted by "sabbasani (via GitHub)" <gi...@apache.org>.
sabbasani commented on PR #13287:
URL: https://github.com/apache/gluten/pull/13287#issuecomment-6493426113
@LuciferYang can merge PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12861] Fix GlutenCachedTableSuite silently losing spark.sql.shuffle.partitions=5 [gluten]
Posted by "sabbasani (via GitHub)" <gi...@apache.org>.
sabbasani commented on PR #13287:
URL: https://github.com/apache/gluten/pull/13287#issuecomment-6513843253
@LuciferYang can merge PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12861] Fix GlutenCachedTableSuite silently losing spark.sql.shuffle.partitions=5 [gluten]
Posted by "sabbasani (via GitHub)" <gi...@apache.org>.
sabbasani commented on code in PR #13287:
URL: https://github.com/apache/gluten/pull/13287#discussion_r4398505459
##########
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/GlutenCachedTableSuite.scala:
##########
@@ -31,12 +31,10 @@ class GlutenCachedTableSuite
with GlutenSQLTestsTrait
with AdaptiveSparkPlanHelper {
import testImplicits._
- // for temporarily disable the columnar table cache globally.
- sys.props.put(GlutenConfig.COLUMNAR_TABLE_CACHE_ENABLED.key, "true")
- override def sparkConf: SparkConf = {
- super.sparkConf.set("spark.sql.shuffle.partitions", "5")
- super.sparkConf.set(GlutenConfig.COLUMNAR_TABLE_CACHE_ENABLED.key, "true")
- }
+ override def sparkConf: SparkConf =
+ super.sparkConf
+ .set("spark.sql.shuffle.partitions", "5")
Review Comment:
@LuciferYang addressed review comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12861] Fix GlutenCachedTableSuite silently losing spark.sql.shuffle.partitions=5 [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13287:
URL: https://github.com/apache/gluten/pull/13287#issuecomment-6462211362
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12861] [TEST] Fix GlutenCachedTableSuite silently losing spark.sql.shuffle.partitions=5 [gluten]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang merged PR #13287:
URL: https://github.com/apache/gluten/pull/13287
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12861] Fix GlutenCachedTableSuite silently losing spark.sql.s… [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13287:
URL: https://github.com/apache/gluten/pull/13287#issuecomment-6176440895
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12861] Fix GlutenCachedTableSuite silently losing spark.sql.s… [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13287:
URL: https://github.com/apache/gluten/pull/13287#issuecomment-6176441648
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12861] Fix GlutenCachedTableSuite silently losing spark.sql.s… [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13287:
URL: https://github.com/apache/gluten/pull/13287#issuecomment-6437360916
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12861] Fix GlutenCachedTableSuite silently losing spark.sql.shuffle.partitions=5 [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13287:
URL: https://github.com/apache/gluten/pull/13287#issuecomment-6462219558
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org