You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "baibaichen (via GitHub)" <gi...@apache.org> on 2026/02/03 05:59:45 UTC
[PR] [GLUTEN-11403][VL] Validate attribute existence before binding in ExpressionConverter [incubator-gluten]
baibaichen opened a new pull request, #11546:
URL: https://github.com/apache/incubator-gluten/pull/11546
## What changes are proposed in this pull request?
This PR fixes a bug where deprecated dataset aggregators (e.g., `typed.sum`) were failing with "Couldn't find value#X in [...]" errors during attribute binding in Spark 4.0+.
**Root Cause**:
This issue is caused by a behavior change in Spark 4.0 and later versions. In Spark 3.5, `BindReferences.bindReference` throws `IllegalStateException` when it cannot find an attribute, while Spark 4.x throws `SparkException`. The original code in `ExpressionConverter` only caught `IllegalStateException`, causing the exception to propagate uncaught in Spark 4.x and preventing proper fallback behavior.
The issue was discovered through Spark UT `DeprecatedDatasetAggregatorSuite`.
**Changes**:
1. Modified `ExpressionConverter.scala` to validate attribute existence **before** attempting to bind, rather than relying on exception catching
2. This approach is more robust and works across all Spark versions
3. Improved error messages for binding failures
4. Added `GlutenDeprecatedDatasetAggregatorSuite` for Spark 4.0 and 4.1 to validate the fix
Fixes #11403
## How was this patch tested?
- Added new test suite `GlutenDeprecatedDatasetAggregatorSuite` for both Spark 4.0 and 4.1
- The test suite includes cases for deprecated dataset aggregators that previously failed
- Enabled the test suites in `VeloxTestSettings` for both Spark versions
## Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11403][VL] Validate attribute existence before binding in ExpressionConverter [incubator-gluten]
Posted by "baibaichen (via GitHub)" <gi...@apache.org>.
baibaichen merged PR #11546:
URL: https://github.com/apache/incubator-gluten/pull/11546
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11403][VL] Validate attribute existence before binding in ExpressionConverter [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11546:
URL: https://github.com/apache/incubator-gluten/pull/11546#issuecomment-3839238565
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11403][VL] Validate attribute existence before binding in ExpressionConverter [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11546:
URL: https://github.com/apache/incubator-gluten/pull/11546#issuecomment-3839327716
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org