[I] Casting `Timestamp(_, None)` to a named timezone fails on DST-ambiguous and nonexistent local times [arrow-rs]

["adriangb (via GitHub)" <gi...@apache.org>]

adriangb opened a new issue, #11037:
URL: https://github.com/apache/arrow-rs/issues/11037

   **Describe the bug**
   
   Casting a `Timestamp(_, None)` array to a `Timestamp(_, Some(tz))` with a **named** timezone fails for every value whose wall-clock reading falls on a daylight saving transition:
   
   - the hour repeated by a "fall back" transition (ambiguous), and
   - the hour skipped by a "spring forward" transition (nonexistent).
   
   With `CastOptions { safe: false }` the whole cast fails with `Cast error: Cannot cast timezone to different timezone`; with `safe: true` the affected values silently become `NULL`. Unambiguous readings and fixed-offset timezones (`+08:00`) are fine.
   
   The cause is `adjust_timestamp_to_timezone` in `arrow-cast/src/cast/mod.rs`:
   
   ```rust
   let adjust = |o| {
       let local = as_datetime::<T>(o)?;
       let offset = to_tz.offset_from_local_datetime(&local).single()?;
       T::from_naive_datetime(local - offset.fix(), None)
   };
   ```
   
   `LocalResult::single()` is `None` for both `LocalResult::Ambiguous` and `LocalResult::None`, so both cases collapse into the failure path.
   
   **To Reproduce**
   
   arrow-cast 59.3.0, arrow-array with the `chrono-tz` feature:
   
   ```rust
   use std::sync::Arc;
   use arrow_array::{Array, ArrayRef, TimestampSecondArray};
   use arrow_cast::{cast_with_options, CastOptions};
   use arrow_schema::{DataType, TimeUnit};
   
   fn main() {
       // Naive wall-clock readings, to be interpreted in America/New_York:
       //   2024-11-01T00:00:00  unambiguous
       //   2024-11-03T01:30:00  ambiguous   (fall back: 01:30 happens twice)
       //   2024-03-10T02:30:00  nonexistent (spring forward: 02:30 never happens)
       let naive: ArrayRef = Arc::new(TimestampSecondArray::from(vec![
           1_730_419_200, 1_730_597_400, 1_710_037_800,
       ]));
       let to = DataType::Timestamp(TimeUnit::Second, Some("America/New_York".into()));
   
       let strict = CastOptions { safe: false, ..Default::default() };
       println!("safe=false: {:?}", cast_with_options(&naive, &to, &strict).map(|a| a.len()));
   
       let lenient = CastOptions { safe: true, ..Default::default() };
       let out = cast_with_options(&naive, &to, &lenient).unwrap();
       let shown: Vec<String> = (0..out.len())
           .map(|i| arrow_cast::display::array_value_to_string(&out, i).unwrap())
           .collect();
       println!("safe=true:  {shown:?}");
   }
   ```
   
   ```
   safe=false: Err(CastError("Cannot cast timezone to different timezone"))
   safe=true:  ["2024-11-01T00:00:00-04:00", "", ""]
   ```
   
   **Expected behavior**
   
   Both values resolve to an instant instead of failing. PostgreSQL 17.11 and DuckDB 1.5.2 agree exactly on how:
   
   ```sql
   SET TimeZone = 'America/New_York';
   SELECT '2024-11-03T01:30:00'::timestamp::timestamptz AS ambiguous,
          '2024-03-10T02:30:00'::timestamp::timestamptz AS nonexistent;
   ```
   
   ```
          ambiguous        |      nonexistent
   ------------------------+------------------------
    2024-11-03 01:30:00-05 | 2024-03-10 03:30:00-04
   ```
   
   - **Ambiguous**: pick the **later** instant, i.e. the post-transition (standard) offset. In chrono terms `LocalResult::Ambiguous(_, later)` → `later`.
   - **Nonexistent**: shift **forward** by the size of the gap, which is the same as interpreting the reading with the pre-transition offset.
   
   So `[1_730_419_200, 1_730_597_400, 1_710_037_800]` should become the instants `[1_730_433_600, 1_730_615_400, 1_710_055_800]`, displayed as `2024-11-01T00:00:00-04:00`, `2024-11-03T01:30:00-05:00`, `2024-03-10T03:30:00-04:00`.
   
   **Additional context**
   
   - Reported against DataFusion as https://github.com/apache/datafusion/issues/25084. I first prototyped a DataFusion-side workaround (https://github.com/apache/datafusion/pull/25115, now closed) but the kernel is the right place: every arrow-rs consumer hits this, and DataFusion has cast paths that call the kernel directly.
   - I will open a PR that changes `adjust_timestamp_to_timezone` to the resolution above. Adding an explicit policy to `CastOptions` (like Arrow C++'s `AssumeTimezoneOptions` with `ambiguous`/`nonexistent` = raise/earliest/latest) would be a breaking change to a public struct, so I am proposing the deterministic behaviour as the fix now; a policy option can follow in a major release if anyone needs `earliest` or `raise`.
   - `string_to_datetime` in `arrow-cast/src/parse.rs` has the same `.single()` pattern in three places (`Error parsing timestamp ...: error computing timezone offset`). That is a separate, generic-`TimeZone` code path and is not part of this issue.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Casting `Timestamp(_, None)` to a named timezone fails on DST-ambiguous and nonexistent local times [arrow-rs]

["Jefffrey (via GitHub)" <gi...@apache.org>]

Jefffrey closed issue #11037: Casting `Timestamp(_, None)` to a named timezone fails on DST-ambiguous and nonexistent local times
URL: https://github.com/apache/arrow-rs/issues/11037


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org