You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Tyagiquamar (via GitHub)" <gi...@apache.org> on 2026/10/01 17:42:47 UTC

[PR] fix: keep the time zone when unparsing timestamp literals [datafusion]

Tyagiquamar opened a new pull request, #25949:
URL: https://github.com/apache/datafusion/pull/25949

   Fixes #25945
   
   ## Root cause
   
   In `Unparser::handle_timestamp` (datafusion/sql/src/unparser/expr.rs), a timestamp `ScalarValue` that carries a time zone is rendered as a tz-aware string via `Dialect::timestamp_with_tz_to_string`, but the wrapping `CAST` was always built with `self.dialect.timestamp_cast_dtype(&time_unit, &None)`: the tz was hard-coded away. The generated SQL was therefore e.g.
   
   ```
   CAST('2024-01-01T11:00:00+08:00' AS TIMESTAMP)
   ```
   
   When PostgreSQL or DuckDB evaluates that SQL with a UTC session zone, the offset in the string is discarded and the value silently shifts by the offset (8 hours here). Unparsing an `Expr::Cast` to the same `Timestamp(_, Some(tz))` type already routed the tz through `timestamp_cast_dtype` and produced `TIMESTAMP WITH TIME ZONE`, so literals and casts disagreed.
   
   ## Fix
   
   Pass the literal's tz through to `Dialect::timestamp_cast_dtype` instead of `&None`, so each dialect renders its own tz-aware timestamp type for literals exactly as it does for casts.
   
   Per-dialect behavior after the fix:
   
   | Dialect | `Timestamp(_, Some(tz))` literal | Unparsed |
   |---|---|---|
   | Default, PostgreSQL, DuckDB, Snowflake, Custom (default builder) | `TIMESTAMP WITH TIME ZONE` (matches what the same dialect emits for a `CAST` with tz) |
   | MySQL (`DATETIME`), SQLite (`TEXT`), BigQuery (`TIMESTAMP`) | unchanged: these dialects intentionally map tz to their naive/own type via `timestamp_cast_dtype` |
   | Custom dialects | fully respected: `with_timestamp_cast_dtype` tz variant now also applies to tz literals |
   
   ## Regression test
   
   The existing `test_timestamp_with_tz_format` and `expr_to_sql_ok` tests pinned the buggy output; their tz-literal expectations are updated to `TIMESTAMP WITH TIME ZONE`. The BigQuery tz-literal cases in `test_timestamp_with_tz_format` keep `TIMESTAMP`, documenting the intended BigQuery dialect behavior; `test_bigquery_dialect_overrides` and the custom-dialect tests are unchanged and still pass.
   
   ## Validation
   
   rustc 1.98.1 (matches rust-toolchain.toml), host Windows:
   
   ```
   cargo test -p datafusion-sql --lib
     test result: ok. 92 passed; 0 failed
   cargo test -p datafusion-sql
     ok. 598 passed; 0 failed  (datafusion-sql integration)
   cargo fmt -p datafusion-sql -- --check   # exit 0
   cargo clippy -p datafusion-sql --all-targets   # no warnings
   ```
   
   `cargo test -p datafusion --lib` fails 118 tests both before and after this change on this machine: all of them panic on missing `ARROW_TEST_DATA`/`PARQUET_TEST_DATA` git submodules (unrelated to the unparser). `datafusion-sql` has no submodule-dependent tests and is the crate touched here.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: keep the time zone when unparsing timestamp literals [datafusion]

Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway commented on PR #25949:
URL: https://github.com/apache/datafusion/pull/25949#issuecomment-6270438294

   @Tyagiquamar Thanks! I took the liberty of adding some additional tests and revising the PR description. Otherwise lgtm.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: keep the time zone when unparsing timestamp literals [datafusion]

Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway merged PR #25949:
URL: https://github.com/apache/datafusion/pull/25949


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: keep the time zone when unparsing timestamp literals [datafusion]

Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway commented on code in PR #25949:
URL: https://github.com/apache/datafusion/pull/25949#discussion_r4230849233


##########
datafusion/sql/src/unparser/expr.rs:
##########
@@ -1261,10 +1261,16 @@ impl Unparser<'_> {
                 .to_string()
         };
 
+        let tz = if tz.is_some() {
+            &Some(Arc::from("+00:00"))
+        } else {
+            &None
+        };

Review Comment:
   Sorry, I don't follow. The PR description says that the fix is to
   ```
   Pass the literal's tz through to Dialect::timestamp_cast_dtype instead of &None
   ```
   But it doesn't seem like that's what we're doing above? I'd have expected that we omit this code and just pass `self.dialect.timestamp_cast_dtype(&time_unit, &tz.cloned())` below.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: keep the time zone when unparsing timestamp literals [datafusion]

Posted by "Tyagiquamar (via GitHub)" <gi...@apache.org>.
Tyagiquamar commented on code in PR #25949:
URL: https://github.com/apache/datafusion/pull/25949#discussion_r4244827492


##########
datafusion/sql/src/unparser/expr.rs:
##########
@@ -1261,10 +1261,16 @@ impl Unparser<'_> {
                 .to_string()
         };
 
+        let tz = if tz.is_some() {
+            &Some(Arc::from("+00:00"))
+        } else {
+            &None
+        };

Review Comment:
   Good catch, thank you! Updated to pass &tz.cloned() directly to self.dialect.timestamp_cast_dtype(&time_unit, &tz.cloned()) without any intermediate synthetic assignment. Verified with cargo fmt --check -p datafusion-sql, cargo clippy -p datafusion-sql, and cargo test -p datafusion-sql --lib unparser::expr::tests in Docker (all 36 unparser expr tests passed).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: keep the time zone when unparsing timestamp literals [datafusion]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #25949:
URL: https://github.com/apache/datafusion/pull/25949#issuecomment-6297002072

   ## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/25949?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `97.56098%` with `2 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 82.93%. Comparing base ([`d89a85d`](https://app.codecov.io/gh/apache/datafusion/commit/d89a85d3e340062a9efad3d8c4b32f10a7b139c5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`5276e8b`](https://app.codecov.io/gh/apache/datafusion/commit/5276e8bef44a2703aa35a3ec3cc1298f26a58adc?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 6 commits behind head on main.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/25949?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [datafusion/sql/src/unparser/expr.rs](https://app.codecov.io/gh/apache/datafusion/pull/25949?src=pr&el=tree&filepath=datafusion%2Fsql%2Fsrc%2Funparser%2Fexpr.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9zcWwvc3JjL3VucGFyc2VyL2V4cHIucnM=) | 97.56% | [1 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25949?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@           Coverage Diff           @@
   ##             main   #25949   +/-   ##
   =======================================
     Coverage   82.93%   82.93%           
   =======================================
     Files        1154     1154           
     Lines      457933   457994   +61     
     Branches   457933   457994   +61     
   =======================================
   + Hits       379786   379848   +62     
   + Misses      55128    55121    -7     
   - Partials    23019    23025    +6     
   ```
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/25949?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org