Re: [I] Using `date_bin` with a time zone in a time range that contains daylight savings does not work [datafusion]

["adriangb (via GitHub)" <gi...@apache.org>]

adriangb commented on issue #10308:
URL: https://github.com/apache/datafusion/issues/10308#issuecomment-5623281951

   Good news on this one: I validated the upstream fix https://github.com/apache/arrow-rs/pull/11038 against your reproducer and it resolves it.
   
   Before, on DataFusion `main` (55.0.0) — note the result column has to be materialized, since wrapping it in `count(*)` lets projection pushdown prune the cast and the error disappears:
   
   ```
   Arrow error: Cast error: Cannot cast timezone to different timezone
   ```
   
   After, with arrow-rs#11038 patched in:
   
   ```
   +---------------------+---------------------------+
   | naive               | brussels                  |
   +---------------------+---------------------------+
   | 2021-03-27T22:30:00 | 2021-03-27T22:30:00+01:00 |
   | 2021-03-28T00:30:00 | 2021-03-28T00:30:00+01:00 |
   | 2021-03-28T01:30:00 | 2021-03-28T01:30:00+01:00 |
   | 2021-03-28T02:30:00 | 2021-03-28T03:30:00+02:00 |   <-- nonexistent local time, shifted forward
   | 2021-03-28T03:30:00 | 2021-03-28T03:30:00+02:00 |
   | 2021-03-28T05:30:00 | 2021-03-28T05:30:00+02:00 |
   +---------------------+---------------------------+
   ```
   
   The offset changes *within* the column, so the resolution is per row rather than one offset applied to the whole array. PostgreSQL 17 and DuckDB/ICU both give `2021-03-28 01:30:00+00` for that nonexistent reading, which matches.
   
   The `date_bin` shape from your original report is also fixed by it, and everything on that path is a `Timestamp(_, None)` -> `Timestamp(_, Some(zone))` cast, so this issue should close once DataFusion picks up an arrow-rs release containing #11038.
   
   For contrast, https://github.com/apache/datafusion/issues/25084 is *not* fully fixed by it — that one also needs https://github.com/apache/arrow-rs/issues/11039, because string literals take a different code path. This issue does not depend on that.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org