You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/10/06 20:29:28 UTC

[I] date_trunc SECOND/MILLISECOND wraps on Spark 4.2, where Spark raises long overflow [datafusion-comet]

andygrove opened a new issue, #6739:
URL: https://github.com/apache/datafusion-comet/issues/6739

   ### Describe the bug
   
   `expressions/datetime/trunc_timestamp.sql` fails on the Spark 4.2 profile in both dictionary variants:
   
   ```
   Error executing SQL 'SELECT date_trunc('SECOND', ts), date_trunc('MILLISECOND', ts)
   FROM test_trunc_ts_overflow' Job aborted due to stage failure: ...
   java.lang.ArithmeticException: long overflow
   	at java.base/java.lang.Math.subtractExact(Math.java:945)
   	at org.apache.spark.sql.catalyst.util.DateTimeUtils$.truncTimestamp(DateTimeUtils.scala:542)
   ```
   
   The query came in with #5956. Its fixture expects Spark to wrap the Long subtraction when SECOND or MILLISECOND truncation of `timestamp_micros(-9223372036854775808)` falls below `Long.MinValue`. Spark 4.1.3 and earlier do wrap. Spark 4.2.0 uses `Math.subtractExact` there and raises `long overflow`, like every other unit (SPARK-56663, apache/spark@da095f9e63). The pull request tier only runs Spark 4.1, so this first showed up on #6664, which has `run-all-spark-profiles`. The nightly will fail the same way.
   
   The fixture is the visible half. Comet's native kernel wraps on every Spark version (`timestamp_trunc_fine_boundary` in `native/spark-expr/src/kernels/temporal.rs`), so on Spark 4.2 Comet returns a timestamp in the year 294247 where Spark raises.
   
   ### Steps to reproduce
   
   ```shell
   ./mvnw test -Pspark-4.2 -Dtest=none -DwildcardSuites=none \
     -Dsuites="org.apache.comet.CometSqlFileTestSuite trunc_timestamp.sql"
   ```
   
   ### Expected behavior
   
   On Spark 4.2, Comet raises `long overflow` when SECOND or MILLISECOND truncation falls below the smallest timestamp. On earlier versions it keeps wrapping, as Spark does.
   
   ### Additional context
   
   Spark 4.1.3 `DateTimeUtils.truncTimestamp`:
   
   ```scala
   case TRUNC_TO_SECOND =>
     micros - Math.floorMod(micros, MICROS_PER_SECOND)
   ```
   
   Spark 4.2.0:
   
   ```scala
   case TRUNC_TO_SECOND =>
     Math.subtractExact(micros, Math.floorMod(micros, MICROS_PER_SECOND))
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] date_trunc SECOND/MILLISECOND wraps on Spark 4.2, where Spark raises long overflow [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #6739: date_trunc SECOND/MILLISECOND wraps on Spark 4.2, where Spark raises long overflow
URL: https://github.com/apache/datafusion-comet/issues/6739


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org