You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2025/10/22 22:12:14 UTC
[I] `trunc` (`TruncDate`) should fall back to Spark for unsupported formats [datafusion-comet]
andygrove opened a new issue, #2633:
URL: https://github.com/apache/datafusion-comet/issues/2633
### Describe the bug
Currently, Comet only supports a subset of formats. Using an unsupported format causes the query to fail. We should fall back to Spark instead.
```sql
SELECT c0, trunc(c0, 'day') from tbl order by c0, c1
```
Fails:
```
org.apache.comet.CometNativeException: InternalError: Unsupported format: "day" for function 'date_trunc'.
```
### Steps to reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `trunc` (`TruncDate`) should fall back to Spark for unsupported formats [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #2633: `trunc` (`TruncDate`) should fall back to Spark for unsupported formats
URL: https://github.com/apache/datafusion-comet/issues/2633
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `trunc` (`TruncDate`) should fall back to Spark for unsupported formats [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on issue #2633:
URL: https://github.com/apache/datafusion-comet/issues/2633#issuecomment-3438400861
These unit tests seem to indicate otherwise -
https://github.com/apache/datafusion-comet/blob/3ad22eb865bbc990b0d9d103833c33f13a7af829/spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala#L688
https://github.com/apache/datafusion-comet/blob/3ad22eb865bbc990b0d9d103833c33f13a7af829/spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala#L657
We should fix this rather than fall back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `trunc` (`TruncDate`) should fall back to Spark for unsupported formats [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #2633:
URL: https://github.com/apache/datafusion-comet/issues/2633#issuecomment-3497347009
> These unit tests seem to indicate otherwise -
>
> [datafusion-comet/spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala](https://github.com/apache/datafusion-comet/blob/3ad22eb865bbc990b0d9d103833c33f13a7af829/spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala#L688)
>
> Line 688 in [3ad22eb](/apache/datafusion-comet/commit/3ad22eb865bbc990b0d9d103833c33f13a7af829)
> test("date_trunc") {
>
> [datafusion-comet/spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala](https://github.com/apache/datafusion-comet/blob/3ad22eb865bbc990b0d9d103833c33f13a7af829/spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala#L671)
>
> Line 671 in [3ad22eb](/apache/datafusion-comet/commit/3ad22eb865bbc990b0d9d103833c33f13a7af829)
> test("trunc with format array") {
>
> We should fix this rather than fall back.
@parthchandra The tests that you linked to only test for valid formats. We did not have any tests for invalid or unsupported formats. This issue demonstrates testing `trunc` with `day` format, which Spark does not support (`day` is supported by `date_trunc` though).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `trunc` (`TruncDate`) should fall back to Spark for unsupported formats [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #2633:
URL: https://github.com/apache/datafusion-comet/issues/2633#issuecomment-3497357721
This issue was fixed by https://github.com/apache/datafusion-comet/pull/2634
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `trunc` (`TruncDate`) should fall back to Spark for unsupported formats [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on issue #2633:
URL: https://github.com/apache/datafusion-comet/issues/2633#issuecomment-3498547266
> > We should fix this rather than fall back.
>
> [@parthchandra](https://github.com/parthchandra) The tests that you linked to only test for valid formats. We did not have any tests for invalid or unsupported formats. This issue demonstrates testing `trunc` with `day` format, which Spark does not support (`day` is supported by `date_trunc` though).
My point remains. Handling the unsupported formats can be added and, imho, preferable to falling back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org