You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2025/10/25 15:38:36 UTC
[I] date_trunc incorrect results in non-UTC timezone [datafusion-comet]
andygrove opened a new issue, #2649:
URL: https://github.com/apache/datafusion-comet/issues/2649
### Describe the bug
PR https://github.com/apache/datafusion-comet/pull/2634 fixed some bugs with `trunc`/`date_trunc`, but I found another bug with the new tests added as part of this PR.
The tests pass when the Spark sessions timezone is `UTC`, but fail for other timezones.
When reading from DataFrame:
```
org.apache.comet.CometNativeException: Fail to process Arrow array with reason: Invalid argument error: RowConverter column schema mismatch, expected Timestamp(Microsecond, Some("America/Denver")) got Timestamp(Microsecond, Some("UTC")).
```
When reading from Parquet:
```
== Results ==
!== Correct Answer - 1000 == == Spark Answer - 1000 ==
struct<c0:timestamp,date_trunc(quarter, c0):timestamp> struct<c0:timestamp,date_trunc(quarter, c0):timestamp>
![3332-12-03 10:00:59.158,3332-09-30 23:00:00.0] [3332-12-03 10:00:59.158,3332-10-01 00:00:00.0]
![3332-12-03 10:04:41.722,3332-09-30 23:00:00.0] [3332-12-03 10:04:41.722,3332-10-01 00:00:00.0]
![3332-12-03 10:26:05.153,3332-09-30 23:00:00.0] [3332-12-03 10:26:05.153,3332-10-01 00:00:00.0]
```
### Steps to reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] date_trunc incorrect results in non-UTC timezone [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #2649:
URL: https://github.com/apache/datafusion-comet/issues/2649#issuecomment-3506840190
short term fix may be to add some casting to UTC
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] date_trunc incorrect results in non-UTC timezone [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #2649:
URL: https://github.com/apache/datafusion-comet/issues/2649#issuecomment-3506791982
just FYI the workaround is
```
CometConf.COMET_EXEC_SHUFFLE_ENABLED.key -> "false"
```
This happens because in RangeRepartition on RowConverter, I'm digging into this real quick to see what is the correct behavior
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] date_trunc incorrect results in non-UTC timezone [datafusion-comet]
Posted by "Licht-T (via GitHub)" <gi...@apache.org>.
Licht-T commented on issue #2649:
URL: https://github.com/apache/datafusion-comet/issues/2649#issuecomment-4274080446
I am looking into this.
> The JVM integration tests in CometTemporalExpressionSuite (which compare Comet output against Spark across UTC, America/Los_Angeles, and Europe/London) are still failing.
Looks this is due to the known limitation of `chrono-tz`: https://github.com/chronotope/chrono-tz/issues/135. It precomputes timezone transitions into generated Rust tables. For zones with recurring future DST rules, those rules are expanded only up to a fixed year range (until 2100) during code generation.
Java `ZoneRules` does not have that limitation. It keeps the recurring transition rules as rules and evaluates them at runtime for arbitrary future years.
I think the solution would be to mark the far-future date as `Incompatible`, falling back to Spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] date_trunc incorrect results in non-UTC timezone [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #2649:
URL: https://github.com/apache/datafusion-comet/issues/2649#issuecomment-4143630178
`TruncTimestamp` (`date_trunc`) is now marked as Incompatible when the timezone is not UTC, which means it falls back to Spark by default for non-UTC timezones. Since default behavior is now correct (via Spark fallback), this is no longer a bug. Relabeling as an enhancement to track implementing correct non-UTC timezone handling in the native implementation in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] date_trunc incorrect results in non-UTC timezone [datafusion-comet]
Posted by "hsiang-c (via GitHub)" <gi...@apache.org>.
hsiang-c commented on issue #2649:
URL: https://github.com/apache/datafusion-comet/issues/2649#issuecomment-3469284635
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] date_trunc incorrect results in non-UTC timezone [datafusion-comet]
Posted by "Licht-T (via GitHub)" <gi...@apache.org>.
Licht-T commented on issue #2649:
URL: https://github.com/apache/datafusion-comet/issues/2649#issuecomment-4274110611
btw, the `chrono-tz` author posted its soft-deprecation and recommends `jiff` as the alternative: https://github.com/chronotope/chrono-tz/issues/231. `jiff` doesn't have the far-future issue, and migrating to it might be an option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] date_trunc incorrect results in non-UTC timezone [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #2649:
URL: https://github.com/apache/datafusion-comet/issues/2649#issuecomment-3469279764
Datafusion regression on `date_trunc` https://github.com/apache/datafusion/issues/18334
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] date_trunc incorrect results in non-UTC timezone [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #2649:
URL: https://github.com/apache/datafusion-comet/issues/2649#issuecomment-3506890456
> I think I have convinced myself that this is only an issue when using `spark.comet.sparkToColumnar.enabled` or other uses of `SparkToColumnar`, so I don't think it is urgent to fix for 0.12.0, but we should at least document this somewhere.
docs PR: https://github.com/apache/datafusion-comet/pull/2740
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] date_trunc incorrect results in non-UTC timezone [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #2649:
URL: https://github.com/apache/datafusion-comet/issues/2649#issuecomment-4173434302
## Findings from PR #3877
I attempted a fix in #3877 that addressed both root causes identified in the issue. Here's a summary of the approach and remaining issues for whoever picks this up next.
### Two bugs identified
**Bug 1 — DST offset preservation during truncation:** The `trunc_date_to_*` functions in `native/spark-expr/src/kernels/temporal.rs` modify date/time fields on a `DateTime<Tz>` while preserving the original UTC offset. When a DST transition falls between the input timestamp and the truncated result (e.g., truncating a November timestamp in `America/Denver` to QUARTER lands on October 1, which is still MDT not MST), the preserved offset is wrong by 1 hour.
**Fix approach:** Extract `naive_local()` from the input `DateTime<Tz>`, perform truncation on the `NaiveDateTime`, then re-apply the timezone via `tz.from_local_datetime()` to get a fresh DST lookup — matching Java's `ZonedDateTime.of()` behavior. The new function `as_micros_from_naive_datetime_tz` replaces `as_micros_from_unix_epoch_utc` and handles:
- `LocalResult::Single` — straightforward
- `LocalResult::Ambiguous` (fall-back) — uses earlier instant, matching Java
- `LocalResult::None` (spring-forward gap) — advances by 1 hour, which assumes all DST gaps are exactly 1 hour
**Bug 2 — Missing timezone annotation on output:** `TimestampTruncExpr::data_type()` in `native/spark-expr/src/datetime_funcs/timestamp_trunc.rs` returned `Timestamp(Microsecond, None)` instead of `Timestamp(Microsecond, Some(tz))`, causing schema mismatches like `RowConverter column schema mismatch` during shuffle/sort. Fix: propagate the expression's timezone into the return type, and annotate the output arrays from `as_timestamp_tz_with_op` with `.with_timezone()`.
### Non-1-hour DST gap limitation
The `LocalResult::None` handling assumes 1-hour DST gaps. Rare timezones like `Australia/Lord_Howe` (30-min DST transition) would produce incorrect results. The PR added a Scala-side `hasNonHourDst()` check using `ZoneId.getRules().getTransitionRules()` to mark those timezones as `Incompatible`, falling back to Spark. This replaced the blanket non-UTC incompatibility.
### What's still failing
The Rust unit tests for the core truncation logic pass (including DST boundary and far-future date tests). The JVM integration tests in `CometTemporalExpressionSuite` (which compare Comet output against Spark across `UTC`, `America/Los_Angeles`, and `Europe/London`) are still failing. I didn't fully diagnose the remaining JVM failures before closing the PR.
### Key files changed
- `native/spark-expr/src/kernels/temporal.rs` — core truncation DST fix
- `native/spark-expr/src/datetime_funcs/timestamp_trunc.rs` — `data_type()` timezone annotation fix
- `spark/src/main/scala/org/apache/comet/serde/datetime.scala` — `hasNonHourDst()` gating logic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] date_trunc incorrect results in non-UTC timezone [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #2649:
URL: https://github.com/apache/datafusion-comet/issues/2649#issuecomment-3503993365
I have a smaller repro:
```scala
test("sort on timestamp after changing session timezone") {
// create data in specific timezone
withSQLConf(SQLConf.SESSION_LOCAL_TIMEZONE.key -> "Europe/London") {
createTimestampTestData.createOrReplaceTempView("tbl")
}
// read data in a different timezone
withSQLConf(SQLConf.SESSION_LOCAL_TIMEZONE.key -> "America/Denver") {
checkSparkAnswerAndOperator("SELECT c0 FROM tbl ORDER BY c0")
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] date_trunc incorrect results in non-UTC timezone [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #2649:
URL: https://github.com/apache/datafusion-comet/issues/2649#issuecomment-3506837662
> just FYI the workaround is
>
> ```
> CometConf.COMET_EXEC_SHUFFLE_ENABLED.key -> "false"
> ```
>
> This happens because in RangeRepartition on RowConverter, I'm digging into this real quick to see what is the correct behavior
Here's what I know so far:
In native execution we have always assumed UTC. When we read from Parquet, we convert to UTC.
When creating arrow schema from Parquet schema we hard-coded UTC. I have a PR https://github.com/apache/datafusion-comet/pull/2734 to remove that hard coding and that is a partial fix.
When native comet scan reads from JVM LocalTableScan it is not always UTC, so we have a mismatch there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] date_trunc incorrect results in non-UTC timezone [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #2649:
URL: https://github.com/apache/datafusion-comet/issues/2649#issuecomment-3506874611
I think I have convinced myself that this is only an issue when using `spark.comet.sparkToColumnar.enabled` or other uses of `SparkToColumnar`, so I don't think it is urgent to fix for 0.12.0, but we should at least document this somewhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org