You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/06/18 14:42:06 UTC
[I] [Bug] unix_timestamp support level disagrees with documented TimestampNTZ tz-conversion divergence [datafusion-comet]
andygrove opened a new issue, #4682:
URL: https://github.com/apache/datafusion-comet/issues/4682
### Describe the bug
`CometUnixTimestamp` (`spark/src/main/scala/org/apache/comet/serde/datetime.scala`) gives inconsistent signals for `TimestampNTZType` input:
- `getUnsupportedReasons` states `TimestampNTZType is not supported because Comet incorrectly applies timezone conversion to TimestampNTZ values`.
- `isSupportedInputType` returns `true` for `TimestampNTZType`, and `getSupportLevel` reports `Compatible()` for it.
So the EXPLAIN-time reason and the runtime behavior disagree. If the native path really does apply session-timezone conversion to `TIMESTAMP_NTZ` (as the documented divergence on `unix_timestamp` in `docs/source/contributor-guide/spark_expressions_support.md` suggests), then Comet produces a different value than Spark for non-UTC session timezones. If it does not, the reason text is stale.
This is the `unix_timestamp` / `UnixTimestamp` analog of the hour/minute/second TimestampNTZ tz-conversion bug tracked in #3180.
### Steps to reproduce
With a non-UTC `spark.sql.session.timeZone`, evaluate `unix_timestamp` (and `to_unix_timestamp`) on a `TIMESTAMP_NTZ` column and compare Comet against Spark.
### Expected behavior
The EXPLAIN-time reason must match runtime behavior. If the native path applies session-timezone conversion to `TIMESTAMP_NTZ`, the support level for that input type should be `Incompatible(Some(...))` (cross-referencing #3180); if it does not, the stale reason text should be removed and the inconsistency resolved.
### Additional context
Split out from #4502 (item 1), surfaced by the `audit-comet-expression` skill run in #4448. Cross-references #3180 (same tz-conversion divergence for hour/minute/second on TimestampNTZ).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] unix_timestamp support level disagrees with documented TimestampNTZ tz-conversion divergence [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #4682: unix_timestamp support level disagrees with documented TimestampNTZ tz-conversion divergence
URL: https://github.com/apache/datafusion-comet/issues/4682
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] unix_timestamp support level disagrees with documented TimestampNTZ tz-conversion divergence [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #4682:
URL: https://github.com/apache/datafusion-comet/issues/4682#issuecomment-5861040500
#4645 removed the stale TimestampNTZ sentence from CometUnixTimestamp, so the reason text now agrees with the Compatible support level and with the native path, which has skipped timezone conversion for NTZ since #4039.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Bug] unix_timestamp support level disagrees with documented TimestampNTZ tz-conversion divergence [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #4682:
URL: https://github.com/apache/datafusion-comet/issues/4682#issuecomment-4836499008
Triage: this is a documentation/consistency issue, not a default-config correctness bug. The native `unix_timestamp` path already handles TimestampNTZ correctly (no timezone conversion) since #4039, and `getSupportLevel` returns `Compatible()` for NTZ to match. The only defect is a stale `getUnsupportedReasons()` string still asserting that Comet "incorrectly applies timezone conversion to TimestampNTZ", which predates #4039. Relabeling to `documentation`/`priority:low`; the fix is to remove/correct that sentence (this differs from #3180 and #2649, where the native path genuinely still diverges).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org