You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "lgbo-ustc (via GitHub)" <gi...@apache.org> on 2026/10/09 11:21:55 UTC
[PR] [GLUTEN-13234][CH] Avoid aggregate topk for identical window keys [gluten]
lgbo-ustc opened a new pull request, #13235:
URL: https://github.com/apache/gluten/pull/13235
CH window top-k aggregate optimization can fail when the window partition keys and order keys are exactly the same expressions.
## What changes are proposed in this pull request?
Fixes #13234.
For queries like:
```sql
ROW_NUMBER() OVER (
PARTITION BY l_returnflag
ORDER BY l_returnflag DESC
)
```
the partition key and order key are identical. This shape is not suitable for the CH window-to-aggregate-topk optimization. When the aggregate top-k path is forced, native execution may cast the generated `rn` column from a wrong source column and fail with errors such as:
```text
Cannot convert NULL value to non-Nullable type: while converting source column `rn#...` to destination column `rn#...`
```
or with a string-to-Int32 cast error when the misplaced source column contains a non-null partition key value.
This patch updates the CH window-to-aggregate-topk rule to skip the aggregate top-k rewrite when the partition keys and order keys are exactly the same expressions. The query will keep the regular window/filter execution path instead.
A regression test is added to `GlutenClickHouseTPCHSaltNullParquetSuite`. The test forces the aggregate-topk optimization to be considered and verifies that this same-key case does not produce `CHAggregateGroupLimitExecTransformer`.
## How was this patch tested?
- Added `GlutenClickHouseTPCHSaltNullParquetSuite` coverage for `row_number` with identical partition/order keys.
- Locally reproduced the failure by forcing:
- `spark.sql.optimizer.windowGroupLimitThreshold=-1`
- `spark.gluten.sql.columnar.backend.ch.runtime_settings.enable_window_group_limit_to_aggregate=true`
- `spark.gluten.sql.columnar.backend.ch.runtime_settings.window.aggregate_topk_high_cardinality_threshold=2.0`
- Verified after the fix that the plan no longer contains `CHAggregateGroupLimitExecTransformer` for the same-key case and the query executes successfully.
- Locally built with:
```bash
./build/mvn -pl backends-clickhouse -am \
-DskipTests \
-Pbackends-clickhouse \
-Pspark-3.5 \
-Pscala-2.13 \
-Pceleborn \
-Pdelta \
-Djava.io.tmpdir=/home/liangjiabiao/w21/tmp \
package
```
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: DeepSeek Harness gpt-5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13234][CH] Avoid aggregate topk for identical window keys [gluten]
Posted by "zzcclp (via GitHub)" <gi...@apache.org>.
zzcclp commented on PR #13235:
URL: https://github.com/apache/gluten/pull/13235#issuecomment-6117051945
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13234][CH] Avoid aggregate topk for identical window keys [gluten]
Posted by "zzcclp (via GitHub)" <gi...@apache.org>.
zzcclp commented on PR #13235:
URL: https://github.com/apache/gluten/pull/13235#issuecomment-6091725694
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13234][CH] Avoid aggregate topk for identical window keys [gluten]
Posted by "zzcclp (via GitHub)" <gi...@apache.org>.
zzcclp merged PR #13235:
URL: https://github.com/apache/gluten/pull/13235
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-13234][CH] Avoid aggregate topk for identical window keys [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13235:
URL: https://github.com/apache/gluten/pull/13235#issuecomment-6080294707
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org