You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/04/28 11:47:39 UTC
[I] Comet native sort lacks row-format support for Struct(Map(...)) sort keys [datafusion-comet]
andygrove opened a new issue, #4123:
URL: https://github.com/apache/datafusion-comet/issues/4123
## Describe the bug
When the sort key is a struct containing a Map, Comet's native sort fails with:
```
org.apache.comet.CometNativeException
Not yet implemented: Row format support not yet implemented for: [SortField {
options: SortOptions { descending: false, nulls_first: true },
data_type: Struct([Field {
name: "data",
data_type: Map(Field { name: "entries", data_type: Struct([
Field { name: "key", data_type: Utf8 },
Field { name: "value", data_type: Utf8 }
]) }, false)
}])
}]
```
This surfaces in Spark 4.1.1's new `having-and-order-by-recursive-type-name-resolution.sql` at query #38:
```sql
SELECT col1.data['key']
FROM VALUES (NAMED_STRUCT('data', MAP('key', 'value', 'num', '42'))) t (col1)
GROUP BY col1
HAVING col1.data['num'] IS NOT NULL
ORDER BY col1.data['key'];
```
## Expected behavior
Comet should fall back to Spark when the sort key includes types not supported by the Arrow row format (Struct/Map combinations are a known gap upstream).
## Workaround
The file is currently disabled when Comet is enabled via `--SET spark.comet.enabled = false` at the top of the file in `dev/diffs/4.1.1.diff`.
## Additional context
PR #4093 enables Spark 4.1.1 in the `Spark SQL Tests` workflow. The underlying limitation lives in `arrow-row` in DataFusion / Arrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet native sort lacks row-format support for Struct(Map(...)) sort keys [datafusion-comet]
Posted by "0lai0 (via GitHub)" <gi...@apache.org>.
0lai0 commented on issue #4123:
URL: https://github.com/apache/datafusion-comet/issues/4123#issuecomment-4344586901
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet native sort lacks row-format support for Struct(Map(...)) sort keys [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #4123: Comet native sort lacks row-format support for Struct(Map(...)) sort keys
URL: https://github.com/apache/datafusion-comet/issues/4123
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet native sort lacks row-format support for Struct(Map(...)) sort keys [datafusion-comet]
Posted by "0lai0 (via GitHub)" <gi...@apache.org>.
0lai0 commented on issue #4123:
URL: https://github.com/apache/datafusion-comet/issues/4123#issuecomment-4335306922
Could I take this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org