You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "akshaytayal (via GitHub)" <gi...@apache.org> on 2026/10/24 10:04:08 UTC
[PR] [GLUTEN-13179][VL] Combine pushed-down filters into a balanced And tree and re-enable GlutenWholeStageCodegenSuite for Spark 4.2 [gluten]
akshaytayal opened a new pull request, #13435:
URL: https://github.com/apache/gluten/pull/13435
## What changes are proposed in this pull request?
`BasicScanExecTransformer` and `FilterExecTransformerBase` recombine the split filter conjuncts with `reduceLeftOption(And)`. That builds a left-deep tree whose depth equals the number of conjuncts, and the recursive `ExpressionConverter` throws `StackOverflowError` for large filters. Spark 4.2's new test `SPARK-56134: Codegen working for empty output` (1000 conjuncts) aborted the whole Spark 4.2 UT group, so #13163 disabled `GlutenWholeStageCodegenSuite` with `@Ignore`.
- Add `FilterHandler.combineConjuncts`, which uses Spark's `PredicateHelper.buildBalancedPredicate`. It keeps the conjunct order, so results are unchanged, and the depth becomes log(N). Use it in the scan and filter transformers.
- Remove the `@Ignore` from the Spark 4.2 `GlutenWholeStageCodegenSuite`.
- The new Spark 4.2 FilterExec CSE tests (SPARK-56431, 3x SPARK-56032) assert on vanilla `WholeStageCodegenExec` and its generated Java, which Gluten replaces. Exclude them and add Gluten variants that check the filter is offloaded to a `WholeStageTransformer`, keeping the result checks with CSE on and off.
Part of #13179 (umbrella #13181).
## How was this patch tested?
Locally on Spark 4.2:
- `GlutenWholeStageCodegenSuite`: 46 succeeded, 0 failed. Before the fix, the run aborted with `StackOverflowError` at SPARK-56134.
- `GlutenParquetV1FilterSuite` / `GlutenParquetV2FilterSuite`: 84 succeeded, 0 failed.
- backends-velox `VeloxScanSuite`, `VeloxTPCHV1Suite`, `VeloxTPCHV2Suite`: 54 succeeded, 0 failed.
CI covers the other Spark versions and backends.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: GitHub Copilot CLI (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13179][VL] Combine pushed-down filters into a balanced And tree and re-enable GlutenWholeStageCodegenSuite for Spark 4.2 [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13435:
URL: https://github.com/apache/gluten/pull/13435#issuecomment-6382873722
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13179][VL] Combine pushed-down filters into a balanced And tree and re-enable GlutenWholeStageCodegenSuite for Spark 4.2 [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13435:
URL: https://github.com/apache/gluten/pull/13435#issuecomment-6383375512
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13179][VL] Combine pushed-down filters into a balanced And tree and re-enable GlutenWholeStageCodegenSuite for Spark 4.2 [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13435:
URL: https://github.com/apache/gluten/pull/13435#issuecomment-6384246928
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13179][VL] Combine pushed-down filters into a balanced And tree and re-enable GlutenWholeStageCodegenSuite for Spark 4.2 [gluten]
Posted by "rui-mo (via GitHub)" <gi...@apache.org>.
rui-mo merged PR #13435:
URL: https://github.com/apache/gluten/pull/13435
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org