You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/10/06 20:29:28 UTC
[I] date_trunc SECOND/MILLISECOND wraps on Spark 4.2, where Spark raises long overflow [datafusion-comet]
andygrove opened a new issue, #6739:
URL: https://github.com/apache/datafusion-comet/issues/6739
### Describe the bug
`expressions/datetime/trunc_timestamp.sql` fails on the Spark 4.2 profile in both dictionary variants:
```
Error executing SQL 'SELECT date_trunc('SECOND', ts), date_trunc('MILLISECOND', ts)
FROM test_trunc_ts_overflow' Job aborted due to stage failure: ...
java.lang.ArithmeticException: long overflow
at java.base/java.lang.Math.subtractExact(Math.java:945)
at org.apache.spark.sql.catalyst.util.DateTimeUtils$.truncTimestamp(DateTimeUtils.scala:542)
```
The query came in with #5956. Its fixture expects Spark to wrap the Long subtraction when SECOND or MILLISECOND truncation of `timestamp_micros(-9223372036854775808)` falls below `Long.MinValue`. Spark 4.1.3 and earlier do wrap. Spark 4.2.0 uses `Math.subtractExact` there and raises `long overflow`, like every other unit (SPARK-56663, apache/spark@da095f9e63). The pull request tier only runs Spark 4.1, so this first showed up on #6664, which has `run-all-spark-profiles`. The nightly will fail the same way.
The fixture is the visible half. Comet's native kernel wraps on every Spark version (`timestamp_trunc_fine_boundary` in `native/spark-expr/src/kernels/temporal.rs`), so on Spark 4.2 Comet returns a timestamp in the year 294247 where Spark raises.
### Steps to reproduce
```shell
./mvnw test -Pspark-4.2 -Dtest=none -DwildcardSuites=none \
-Dsuites="org.apache.comet.CometSqlFileTestSuite trunc_timestamp.sql"
```
### Expected behavior
On Spark 4.2, Comet raises `long overflow` when SECOND or MILLISECOND truncation falls below the smallest timestamp. On earlier versions it keeps wrapping, as Spark does.
### Additional context
Spark 4.1.3 `DateTimeUtils.truncTimestamp`:
```scala
case TRUNC_TO_SECOND =>
micros - Math.floorMod(micros, MICROS_PER_SECOND)
```
Spark 4.2.0:
```scala
case TRUNC_TO_SECOND =>
Math.subtractExact(micros, Math.floorMod(micros, MICROS_PER_SECOND))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] date_trunc SECOND/MILLISECOND wraps on Spark 4.2, where Spark raises long overflow [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #6739: date_trunc SECOND/MILLISECOND wraps on Spark 4.2, where Spark raises long overflow
URL: https://github.com/apache/datafusion-comet/issues/6739
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org