You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/09/23 17:06:52 UTC
[I] CometSort sorts a multi-column key containing a collated string by raw bytes [datafusion-comet]
andygrove opened a new issue, #6158:
URL: https://github.com/apache/datafusion-comet/issues/6158
### Describe the bug
`QueryPlanSerde.supportedSortType` only type-checks a sort that has a single sort order. A multi-column sort whose key includes a string with a non-default collation (Spark 4.0+) therefore converts to `CometSortExec`, which sorts it by raw bytes rather than by the collation.
The collation check in `CometShuffleExchangeExec.columnarShuffleFailureReasons` keeps the stage off Comet for hash and range partitioning, which #6110 found is the only thing stopping this there. `SinglePartition` and round-robin exchanges have no such check, so the sort still reaches native.
### Steps to reproduce
On `main` at cccc08b7c, default Spark 4.1 profile:
```sql
CREATE TABLE repro_coll_sort (_1 INT, _2 STRING) USING parquet;
INSERT INTO repro_coll_sort VALUES (1, 'b'), (2, 'A'), (3, 'a'), (4, 'B');
SELECT _1, row_number() OVER (ORDER BY c, _1) AS rn
FROM (SELECT _1, _2 COLLATE UTF8_LCASE AS c FROM repro_coll_sort);
```
The executed plan has `CometWindowExec` over `CometSort [c ASC NULLS FIRST, _1 ASC NULLS FIRST]` over a `SinglePartition` `CometColumnarExchange`, and the row numbers differ:
| `_1` | Spark `rn` | Comet `rn` |
|---|---|---|
| 1 | 3 | 4 |
| 2 | 1 | 1 |
| 3 | 2 | 3 |
| 4 | 4 | 2 |
Comet orders the rows `A, B, a, b`, which is byte order. Spark orders them `A, a, b, B`, which is `UTF8_LCASE` order with `_1` breaking the ties.
### Expected behavior
`CometSortExec` should decline a sort key that contains a non-`UTF8_BINARY` collated string at any position or nesting depth, or route it through the codegen dispatcher, so the result matches Spark.
### Additional context
Found while reviewing #6110. There, removing the shuffle collation check makes `listagg(DISTINCT ...)` under `utf8_lcase` return `aabb` instead of `ab`, because `CometSort` stays on a two-column collated key. The #5302 author reported the same limitation, including the window order-spec variant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CometSort sorts a multi-column key containing a collated string by raw bytes [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao closed issue #6158: CometSort sorts a multi-column key containing a collated string by raw bytes
URL: https://github.com/apache/datafusion-comet/issues/6158
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CometSort sorts a multi-column key containing a collated string by raw bytes [datafusion-comet]
Posted by "LinSimon-901101 (via GitHub)" <gi...@apache.org>.
LinSimon-901101 commented on issue #6158:
URL: https://github.com/apache/datafusion-comet/issues/6158#issuecomment-5818255976
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org