You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "baibaichen (via GitHub)" <gi...@apache.org> on 2026/03/09 10:12:21 UTC
[PR] [GLUTEN-11550][VL] Enable GlutenDataFrameSubquerySuite for Spark 4.1 [incubator-gluten]
baibaichen opened a new pull request, #11727:
URL: https://github.com/apache/incubator-gluten/pull/11727
## What changes are proposed in this pull request?
Fix struct join key validation and enable `GlutenDataFrameSubquerySuite` for Spark 4.1.
Spark 4.1 added `isin(Dataset)` API which creates struct IN subquery predicates. These are converted to `BroadcastHashJoin` with struct-typed join keys where field names may differ between sides (e.g., `struct(a, b)` vs `struct(c, d)`).
**Changes:**
1. **`JoinExecTransformer.scala`** — Remove struct field name comparison in `sameType()` to align with Spark's `DataType.equalsStructurally(ignoreNullability=true)` semantics. Spark's native `HashJoin` only checks structural type compatibility (field count + field types by position), not field names. This allows struct IN subqueries to be offloaded to Velox natively.
2. **`Validators.scala`** — Add try-catch in `FallbackByNativeValidation.offloadAttempt` to catch exceptions during transformer construction and return graceful fallback instead of crashing.
3. **`VeloxTestSettings.scala`** — Enable `GlutenDataFrameSubquerySuite` for Spark 4.1 (was previously disabled with `// TODO: 4.x`).
## How was this patch tested?
- `GlutenDataFrameSubquerySuite` on Spark 4.1: all 47 tests passed (struct IN subqueries offloaded to Velox natively)
- `GlutenDataFrameSubquerySuite` on Spark 4.0: all 40 tests passed (no regression)
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: GitHub Copilot
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11550][VL] Enable GlutenDataFrameSubquerySuite for Spark 4.1 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11727:
URL: https://github.com/apache/incubator-gluten/pull/11727#issuecomment-4029248768
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11550][VL] Enable GlutenDataFrameSubquerySuite for Spark 4.1 [incubator-gluten]
Posted by "baibaichen (via GitHub)" <gi...@apache.org>.
baibaichen commented on PR #11727:
URL: https://github.com/apache/incubator-gluten/pull/11727#issuecomment-4031620263
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11550][VL] Enable GlutenDataFrameSubquerySuite for Spark 4.1 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11727:
URL: https://github.com/apache/incubator-gluten/pull/11727#issuecomment-4024092852
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11550][VL] Enable GlutenDataFrameSubquerySuite for Spark 4.1 [gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan merged PR #11727:
URL: https://github.com/apache/gluten/pull/11727
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11550][VL] Enable GlutenDataFrameSubquerySuite for Spark 4.1 [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11727:
URL: https://github.com/apache/gluten/pull/11727#issuecomment-4038736117
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11550][VL] Enable GlutenDataFrameSubquerySuite for Spark 4.1 [gluten]
Posted by "baibaichen (via GitHub)" <gi...@apache.org>.
baibaichen commented on PR #11727:
URL: https://github.com/apache/gluten/pull/11727#issuecomment-4035226118
@zhouyuan please review this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11550][VL] Enable GlutenDataFrameSubquerySuite for Spark 4.1 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11727:
URL: https://github.com/apache/incubator-gluten/pull/11727#issuecomment-4022660137
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org