You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "akshaytayal (via GitHub)" <gi...@apache.org> on 2026/10/12 03:27:37 UTC
[PR] [GLUTEN-13174][VL] Re-enable the Spark 4.2 storage-partitioned join tests [gluten]
akshaytayal opened a new pull request, #13243:
URL: https://github.com/apache/gluten/pull/13243
## What changes are proposed in this pull request?
Re-enables the 29 Spark 4.2 storage-partitioned join tests in `GlutenKeyGroupedPartitioningSuite` that #13163 disabled.
This first commit only removes the 29 `.exclude(...)` entries from `gluten-ut/spark42/.../VeloxTestSettings.scala`, so the Spark 4.2 UT CI shows which of them still fail on current `main`. This PR will then fix what it can and skip the rest again, with follow-up PRs for those (see #13139 and the reviewer notes in #13174).
Part of #13174.
## How was this patch tested?
Spark 4.2 UT CI (`spark-test-spark42` group 3 runs `org.apache.spark.sql.connector`).
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: GitHub Copilot CLI (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13174][VL] Re-enable the Spark 4.2 storage-partitioned join tests [gluten]
Posted by "baibaichen (via GitHub)" <gi...@apache.org>.
baibaichen merged PR #13243:
URL: https://github.com/apache/gluten/pull/13243
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13174][VL] Re-enable the Spark 4.2 storage-partitioned join tests [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13243:
URL: https://github.com/apache/gluten/pull/13243#issuecomment-6117540905
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13174][VL] Re-enable the Spark 4.2 storage-partitioned join tests [gluten]
Posted by "weiting-chen (via GitHub)" <gi...@apache.org>.
weiting-chen commented on code in PR #13243:
URL: https://github.com/apache/gluten/pull/13243#discussion_r4271688056
##########
gluten-ut/spark42/src/test/scala/org/apache/spark/sql/connector/GlutenKeyGroupedPartitioningSuite.scala:
##########
@@ -359,8 +370,11 @@ class GlutenKeyGroupedPartitioningSuite
val shuffles = collectColumnarShuffleExchangeExec(df.queryExecution.executedPlan)
Review Comment:
**Optional: check both shuffle implementations below the join**
**Target Location:** `gluten-ut/spark42/src/test/scala/org/apache/spark/sql/connector/GlutenKeyGroupedPartitioningSuite.scala:370-379`, and the analogous restored SPJ copies.
**Problem:** The assertion says there is no shuffle, but this collector sees only columnar exchanges. In the `pushDownValues=false` iteration, the grouping check is skipped entirely: a future fallback could introduce a vanilla `ShuffleExchangeExec` below the join, preserve the answer, and still pass this absence check. This is a **nonblocking, inherited test-oracle gap**, not evidence of a current extra shuffle or wrong result. Confidence: high.
**Evidence:**
```scala
val shuffles = collectColumnarShuffleExchangeExec(df.queryExecution.executedPlan)
assert(shuffles.isEmpty, "should not contain any shuffle")
```
The updated grouping assertions run only under `if (pushDownValues)`. The existing `collectShuffles` override at lines 92-101 already handles both Spark and Gluten shuffle exchanges beneath either join representation.
**Suggested Fix:**
```scala
val shuffles = collectShuffles(df.queryExecution.executedPlan)
assert(shuffles.isEmpty, "should not contain any shuffle")
```
Use the **join-scoped** helper, rather than global `collectAllShuffles`: the `ORDER BY` outside the join can legitimately need its own exchange. Keep explicitly native-only checks elsewhere unchanged. Apply the same substitution to the analogous restored no-shuffle assertions and rerun those SPJ variants.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13174][VL] Re-enable the Spark 4.2 storage-partitioned join tests [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13243:
URL: https://github.com/apache/gluten/pull/13243#issuecomment-6117205677
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13174][VL] Re-enable the Spark 4.2 storage-partitioned join tests [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13243:
URL: https://github.com/apache/gluten/pull/13243#issuecomment-6117748128
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org