You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/02/11 23:04:45 UTC
[PR] fix: [df52] route timestamp timezone mismatches through spark_parquet_convert [datafusion-comet]
andygrove opened a new pull request, #3494:
URL: https://github.com/apache/datafusion-comet/pull/3494
## Summary
- Fix incorrect timestamp timezone handling in the schema adapter on the df52 branch
- INT96 Parquet timestamps coerced to `Timestamp(us, None)` by DataFusion were being routed through Spark's `Cast` expression when the logical schema expected `Timestamp(us, Some("UTC"))`. Spark's Cast treats `None`-timezone as TimestampNTZ (local time) and applies a timezone conversion, shifting values by the session timezone offset (e.g., -5h45m for Asia/Kathmandu)
- Route `Timestamp -> Timestamp` mismatches through `CometCastColumnExpr` which delegates to `spark_parquet_convert`, handling this as a metadata-only timezone relabel
## Test plan
- [ ] Existing test "SortMergeJoin with unsupported key type should fall back to Spark" in `CometJoinSuite` should now pass
- [ ] Existing Rust tests (`parquet_roundtrip_int_as_string`, `parquet_roundtrip_unsigned_int`) continue to pass
- [ ] CI passes
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: [df52] route timestamp timezone mismatches through spark_parquet_convert [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #3494:
URL: https://github.com/apache/datafusion-comet/pull/3494
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: [df52] route timestamp timezone mismatches through spark_parquet_convert [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #3494:
URL: https://github.com/apache/datafusion-comet/pull/3494#issuecomment-3891079334
## CI Comparison: PR #3494 vs df52 baseline (PR #3470)
| | PR #3470 (df52 baseline) | PR #3494 (with this fix) |
|---|---|---|
| **Failing CI jobs** | 34 | 30 |
| **Passing CI jobs** | same set + 4 extra failures | same set |
### Jobs fixed by this PR (fail in #3470, pass in #3494)
- `spark-sql-native_datafusion-sql_core-3/spark-3.5.8`
- `spark-sql-native_datafusion-sql_hive-2/spark-3.5.8`
- `ubuntu-latest/Spark 3.5, JDK 17, Scala 2.12/native_datafusion [exec]`
- `ubuntu-latest/Spark 3.5, JDK 17, Scala 2.12/native_datafusion [sql]`
### Regressions introduced by this PR (pass in #3470, fail in #3494)
None — every failure in #3494 also exists in #3470.
### Summary
This PR is a strict improvement over the df52 baseline: **4 fewer failing CI jobs with zero regressions**.
---
> [!NOTE]
> This comment was generated with the assistance of AI (Claude Code) and should be verified independently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org