You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/09/28 19:50:14 UTC

[PR] fix: truncate timestamps around DST transitions the way Spark does [datafusion-comet]

andygrove opened a new pull request, #6354:
URL: https://github.com/apache/datafusion-comet/pull/6354

   ## Which issue does this PR close?
   
   Closes #5633.
   
   ## Rationale for this change
   
   The timezone-aware path of the native `date_trunc` built the truncated time with chrono's `with_hour`, `with_minute`, `with_day0` and `with_month0` on a `DateTime<Tz>`. Those return `None` when the local result is ambiguous (fall-back) or falls in a gap (spring-forward). `as_micros_from_unix_epoch_utc` then unwrapped the `None`, so truncating a timestamp near a DST transition panicked. `trunc_timestamp_dst_ambiguous.sql` had every query marked `ignore` because of this.
   
   Even where it didn't panic, the old path didn't follow Spark for ambiguous results. Spark's `DateTimeUtils.truncTimestamp` treats the levels differently:
   
   | Level | Spark | Ambiguous local result | Local result in a gap |
   | --- | --- | --- | --- |
   | `MICROSECOND`, `MILLISECOND`, `SECOND` | truncates the instant (offsets are whole seconds) | n/a | n/a |
   | `MINUTE`, `HOUR`, `DAY` | `ZonedDateTime.truncatedTo` | keeps the input's offset if it is still valid | moved forward by the gap's length |
   | `WEEK`, `MONTH`, `QUARTER`, `YEAR` | truncate the local date, then `LocalDate.atStartOfDay` | earlier offset | first instant after the gap |
   
   So on the 2024-11-03 fall-back in `America/Los_Angeles`, the second 01:30 (PST) truncates to 01:00 PST, not to the first 01:00 (PDT).
   
   ## What changes are included in this PR?
   
   - The timezone-aware truncation now works on the local time. It truncates the naive local datetime with the same helpers the `TIMESTAMP_NTZ` path uses, then resolves it back to an instant with the rule for its level from the table above. A gap takes the offset from before the gap through `resolve_local_datetime`, the helper the date-to-timestamp cast already uses. That gives the same instant as Java's rule.
   - `MICROSECOND`, `MILLISECOND` and `SECOND` truncate the instant directly, as Spark does, instead of going through the calendar.
   - `as_micros_from_unix_epoch_utc` and the `DateTime<Tz>`-based helpers are removed.
   - A new Rust unit test checks 49 truncations against values computed with `java.time`, which is what Spark's `truncTimestamp` calls. It covers both 01:30s of the 2024-11-03 Los Angeles fall-back, the 2024-03-10 spring-forward, the 1970-10-25 repro from the issue, São Paulo's skipped midnight on 2018-11-04, and both 23:30s of its 2019-02-16 fall-back.
   - `trunc_timestamp_dst_ambiguous.sql` runs its six queries again. It adds the second occurrence of the ambiguous hour and a `MINUTE` query.
   - A new `trunc_timestamp_dst_midnight.sql` covers São Paulo's midnight transitions.
   
   This conflicts with #5956, which also reworks this kernel. Truncation in non-UTC sessions still defaults to the codegen dispatcher, since it remains `Incompatible` because of the chrono-tz horizon and #6331. This makes the opt-in native path correct.
   
   @coderfender, I picked this up as part of the timezone EPIC (#6335). Happy to hand it back if you already have work in progress.
   
   ## How are these changes tested?
   
   - **Old kernel:** the new unit test panics at `temporal.rs:166`. Both `trunc_timestamp_dst_ambiguous.sql` and `trunc_timestamp_dst_midnight.sql` fail with the native panic.
   - **With this change:**
     - Both SQL files match Spark.
     - All `expressions/datetime/` SQL file tests and `CometTemporalExpressionSuite` pass on Spark 4.1.
     - The new and existing kernel unit tests pass.
     - `cargo clippy --all-targets -- -D warnings` is clean for `datafusion-comet-spark-expr`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: truncate timestamps around DST transitions the way Spark does [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #6354:
URL: https://github.com/apache/datafusion-comet/pull/6354#discussion_r4127069931


##########
native/spark-expr/src/kernels/temporal.rs:
##########
@@ -101,80 +101,71 @@ fn trunc_days_to_week(days: i32) -> Option<i32> {
     Some(days - days_since_monday)
 }
 
-// Based on arrow_arith/temporal.rs:extract_component_from_datetime_array
-// Transforms an array of DateTime<Tz> to an array of TimestampMicrosecond after applying an
-// operation. The output array carries the input timezone annotation so downstream operators
-// (shuffle, sort, row converter) observe a matching schema.
-fn as_timestamp_tz_with_op<A: ArrayAccessor<Item = T::Native>, T: ArrowTemporalType, F>(
-    iter: ArrayIter<A>,
-    mut builder: PrimitiveBuilder<TimestampMicrosecondType>,
-    tz_str: &str,
-    op: F,
-) -> Result<TimestampMicrosecondArray, SparkError>
-where
-    F: Fn(DateTime<Tz>) -> i64,
-    i64: From<T::Native>,
-{
-    let tz: Tz = tz_str.parse()?;
-    for value in iter {
-        match value {
-            Some(value) => match as_datetime_with_timezone::<T>(value.into(), tz) {
-                Some(time) => builder.append_value(op(time)),
-                _ => {
-                    return Err(SparkError::Internal(
-                        "Unable to read value as datetime".to_string(),
-                    ));
-                }
-            },
-            None => builder.append_null(),
+/// How `date_trunc` truncates a timestamp with a timezone. Spark's `DateTimeUtils.truncTimestamp`
+/// treats the levels differently, and matching it matters around DST transitions.
+#[derive(Clone, Copy)]
+enum TzTrunc {
+    /// `MICROSECOND`, `MILLISECOND` and `SECOND`. Offsets are whole seconds, so Spark truncates the
+    /// instant itself. The value is the unit in microseconds.
+    Instant(i64),
+    /// `MINUTE`, `HOUR` and `DAY`. Spark uses `ZonedDateTime.truncatedTo`, which truncates the local
+    /// time and keeps the input's offset if the result is ambiguous.
+    LocalTime(NtzTruncFn),
+    /// `WEEK`, `MONTH`, `QUARTER` and `YEAR`. Spark truncates the local date and then takes
+    /// `LocalDate.atStartOfDay`, which uses the earlier offset if midnight is ambiguous.
+    LocalDate(NtzTruncFn),
+}
+
+/// Truncates `micros` in `tz` the way Spark's `DateTimeUtils.truncTimestamp` does. A truncated
+/// local time that falls in a DST gap takes the offset from before the gap, which gives the same
+/// instant as Java moving it forward by the gap's length. For the date levels that is also where
+/// `atStartOfDay` puts a day whose midnight falls in a gap that starts at midnight. Returns `None`
+/// if `micros` is out of chrono's range.
+fn trunc_timestamp_in_tz(micros: i64, tz: &Tz, trunc: TzTrunc) -> Option<i64> {
+    let (trunc_fn, keep_offset) = match trunc {
+        TzTrunc::Instant(unit) => return Some(micros - micros.rem_euclid(unit)),
+        TzTrunc::LocalTime(trunc_fn) => (trunc_fn, true),
+        TzTrunc::LocalDate(trunc_fn) => (trunc_fn, false),
+    };
+    let utc = DateTime::from_timestamp_micros(micros)?.naive_utc();
+    let input_offset = tz.offset_from_utc_datetime(&utc).fix();
+    let local = trunc_fn(utc.checked_add_offset(input_offset)?)?;
+    let truncated = match tz.offset_from_local_datetime(&local) {
+        LocalResult::Single(offset) => local.checked_sub_offset(offset.fix())?,
+        LocalResult::Ambiguous(earlier, later) => {
+            let offset = if keep_offset && later.fix() == input_offset {
+                later
+            } else {
+                earlier
+            };
+            local.checked_sub_offset(offset.fix())?
         }
-    }
-    Ok(builder.finish().with_timezone(tz_str))
+        LocalResult::None => resolve_local_datetime(tz, local).naive_utc(),

Review Comment:
   [P2] Resolve `LocalDate` gaps at the first valid instant. With session timezone `America/Toronto` and `spark.comet.expression.TruncTimestamp.allowIncompatible=true`, apply `date_trunc('WEEK', ts)` to a timestamp column containing `1919-03-31 00:45:00-04:00`. Spark returns `1919-03-31 00:30:00-04:00`, but this branch returns `01:00:00-04:00`. Toronto's gap ran from the previous day's 23:30 to 00:30. `resolve_local_datetime` shifts midnight by the full gap length, whereas Spark's `LocalDate.atStartOfDay` selects the gap's end. This replaces the base's panic with silently incorrect weekly boundaries, even producing a result later than this input. Keep gap-length shifting for `LocalTime`, but resolve `LocalDate` to the transition boundary and add this regression case.
   
   Evidence: Compiled the unmodified head/base kernels against locked Arrow 59.3.0 and chrono 0.4.45. For a timezone-annotated TimestampMicrosecondArray containing -1601752500000000, timestamp_trunc(..., "WEEK") returns -1601751600000000 at head. The base panics. Java 21's LocalDate.of(1919, 3, 31).atStartOfDay(ZoneId.of("America/Toronto")) returns -1601753400000000, corresponding to 00:30-04:00. Spark's date-level truncation uses this atStartOfDay rule through daysToMicros across the checked supported versions.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: truncate timestamps around DST transitions the way Spark does [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #6354:
URL: https://github.com/apache/datafusion-comet/pull/6354#issuecomment-5997662278

   Thanks @sunchao, @parthchandra and @comphead. You're right about the date-level gap. `resolve_local_datetime` shifts midnight by the length of the gap, which only matches `LocalDate.atStartOfDay` when the gap starts exactly at midnight, so Toronto's 1919-03-31 comes out as 01:00 instead of 00:30.
   
   sam-1112's #5956 rewrites the same kernel for literal formats. It already resolves a date-level gap to the transition instant and takes the earlier midnight in an overlap, with Toronto, Asuncion and Havana tests. Rather than fix the same thing twice and keep the two PRs conflicting, I'd like #5956 to land first. After it, the only path that still panics on DST transitions is the per-row format path (`timestamp_trunc_array_fmt_helper`, through `as_micros_from_unix_epoch_utc`). I'll then cut this PR down to that path, reuse the helpers from #5956, and pick up comphead's suggestions there. I'm moving this to draft until then.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: truncate per-row date_trunc formats with the literal-format kernel [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #6354:
URL: https://github.com/apache/datafusion-comet/pull/6354#issuecomment-6042986186

   I've reworked this on top of #5956. Instead of carrying its own truncation rules, the path for a format column now groups the rows by format and runs each group through the literal-format kernel. The Toronto date-level gap @sunchao and @parthchandra found is handled by the `atStartOfDay` gap search from #5956, and @comphead's Havana case is in both DST fixtures. The code those inline comments were on is gone, so they no longer apply. The description has the details and the test results.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: truncate per-row date_trunc formats with the literal-format kernel [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #6354:
URL: https://github.com/apache/datafusion-comet/pull/6354#discussion_r4231276868


##########
native/spark-expr/src/datetime_funcs/timestamp_trunc.rs:
##########
@@ -133,7 +133,7 @@ impl PhysicalExpr for TimestampTruncExpr {
                 Ok(ColumnarValue::Array(relabel(result)?))
             }
             (ColumnarValue::Array(ts), ColumnarValue::Array(formats)) => {
-                let result = timestamp_trunc_array_fmt_dyn(&resolve_tz(ts)?, &formats)?;
+                let result = timestamp_trunc_array_fmt_dyn(&resolve_tz(ts)?, &formats, wrap)?;

Review Comment:
   Done in 97728ba12. A NULL literal format now returns NULL: an array of the input's length, or a NULL scalar when the timestamp is a scalar too. A literal timestamp with a format column is expanded with `to_array_of_size` and goes through `timestamp_trunc_array_fmt_dyn`. `null_literal_format_gives_null` and `literal_timestamp_with_a_format_column` cover both.



##########
spark/src/test/resources/sql-tests/expressions/datetime/trunc_timestamp_dst_format_column.sql:
##########
@@ -0,0 +1,66 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements.  See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership.  The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License.  You may obtain a copy of the License at
+--
+--   http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied.  See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+
+-- date_trunc with the format in a column, around DST overlaps, gaps and historical offsets. Each
+-- row is truncated with the rules a literal format applies (see trunc_timestamp_dst_ambiguous.sql
+-- for what each zone covers), and a NULL format gives NULL. A non-literal format is Incompatible,
+-- because an invalid format throws instead of returning NULL.
+
+-- Config: spark.comet.expression.TruncTimestamp.allowIncompatible=true
+-- Config: spark.sql.parquet.int96RebaseModeInWrite=CORRECTED
+-- ConfigMatrix: spark.sql.session.timeZone=UTC,America/Los_Angeles,America/New_York,America/Sao_Paulo,Africa/Monrovia,Asia/Aden,America/Havana,America/Toronto,America/Asuncion
+
+statement
+CREATE TABLE test_trunc_fmt_ts(ts timestamp) USING parquet
+
+statement
+INSERT INTO test_trunc_fmt_ts VALUES
+  (timestamp('1919-04-02T16:00:00Z')),
+  (timestamp('1919-03-31T04:30:00Z')),
+  (timestamp('1919-03-31T04:45:00Z')),
+  (timestamp('2023-10-15T12:00:00Z')),
+  (TIMESTAMP '1960-06-15 10:30:45'),
+  (timestamp('1947-03-13T20:53:30.123Z')),
+  (timestamp('1972-01-07T00:44:45Z')),
+  (timestamp('2018-11-04T02:30:15.123456Z')),
+  (timestamp('2018-11-04T03:30:15.123456Z')),
+  (timestamp('2024-03-10T09:30:15.123456Z')),
+  (timestamp('2024-03-10T10:30:15.123456Z')),
+  (timestamp('2024-11-03T08:30:15.123456Z')),
+  (timestamp('2024-11-03T09:30:15.123456Z')),
+  (timestamp('2020-11-01T04:30:00Z')),
+  (timestamp('2020-11-01T05:30:00Z')),
+  (timestamp('2020-11-15T12:00:00Z')),
+  (NULL)
+
+statement
+CREATE TABLE test_trunc_fmt_unit(fmt string) USING parquet
+
+statement
+INSERT INTO test_trunc_fmt_unit VALUES
+  ('YEAR'), ('yyyy'), ('YY'), ('QUARTER'), ('MONTH'), ('mon'), ('MM'), ('WEEK'), ('DAY'), ('dd'),
+  ('HOUR'), ('MINUTE'), ('SECOND'), ('MILLISECOND'), ('MICROSECOND'), (NULL)
+
+statement
+CREATE TABLE test_trunc_fmt(ts timestamp, fmt string) USING parquet
+
+statement
+INSERT INTO test_trunc_fmt SELECT ts, fmt FROM test_trunc_fmt_ts CROSS JOIN test_trunc_fmt_unit
+
+query expect_native(date_trunc)
+SELECT ts, fmt, date_trunc(fmt, ts) FROM test_trunc_fmt ORDER BY ts, fmt

Review Comment:
   Added both, `date_trunc(NULL, ts)` and `date_trunc(fmt, TIMESTAMP '2024-11-03 01:30:00')`, under `expect_native(date_trunc)` in all nine sessions. With the two new match arms removed, the file fails in every session with `Invalid input to function TimestampTrunc`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: truncate per-row date_trunc formats with the literal-format kernel [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #6354:
URL: https://github.com/apache/datafusion-comet/pull/6354#discussion_r4220229600


##########
native/spark-expr/src/kernels/temporal.rs:
##########
@@ -1260,154 +1213,109 @@ pub(crate) fn timestamp_trunc_array_fmt_dyn(
     array: &dyn Array,
     formats: &dyn Array,
 ) -> Result<ArrayRef, SparkError> {
-    match (array.data_type().clone(), formats.data_type().clone()) {
-        (DataType::Dictionary(_, _), DataType::Dictionary(_, _)) => {
-            downcast_dictionary_array!(
-                formats => {
-                    downcast_dictionary_array!(
-                        array => {
-                            timestamp_trunc_array_fmt_dict_dict(
-                                    &array.downcast_dict::<TimestampMicrosecondArray>().unwrap(),
-                                    &formats.downcast_dict::<StringArray>().unwrap())
-                            .map(|a| Arc::new(a) as ArrayRef)
-                        }
-                        dt => return_compute_error_with!("timestamp_trunc does not support", dt)
-                    )
-                }
-                fmt => return_compute_error_with!("timestamp_trunc does not support format type", fmt),
-            )
+    let array = unpack_dictionary(array)?;
+    let formats = unpack_dictionary(formats)?;
+    let array = match array.data_type() {
+        DataType::Timestamp(TimeUnit::Microsecond, _) => {
+            array.as_primitive::<TimestampMicrosecondType>()
         }
-        (DataType::Dictionary(_, _), DataType::Utf8) => {
-            downcast_dictionary_array!(
-                array => {
-                  timestamp_trunc_array_fmt_dict_plain(
-                        &array.downcast_dict::<PrimitiveArray<TimestampMicrosecondType>>().unwrap(),
-                        formats.as_any().downcast_ref::<StringArray>()
-                            .expect("Unexpected value type in formats"))
-                  .map(|a| Arc::new(a) as ArrayRef)
-                }
-                dt => return_compute_error_with!("timestamp_trunc does not support", dt),
-            )
-        }
-        (DataType::Timestamp(TimeUnit::Microsecond, _), DataType::Dictionary(_, _)) => {
-            downcast_dictionary_array!(
-                formats => {
-                downcast_temporal_array!(array => {
-                        timestamp_trunc_array_fmt_plain_dict(
-                                array,
-                                &formats.downcast_dict::<StringArray>().unwrap())
-                        .map(|a| Arc::new(a) as ArrayRef)
-                    }
-                    dt => return_compute_error_with!("timestamp_trunc does not support", dt),
-                    )
-                }
-                fmt => return_compute_error_with!("timestamp_trunc does not support format type", fmt),
-            )
-        }
-        (DataType::Timestamp(TimeUnit::Microsecond, _), DataType::Utf8) => {
-            downcast_temporal_array!(
-                array => {
-                    timestamp_trunc_array_fmt_plain_plain(array,
-                        formats.as_any().downcast_ref::<StringArray>().expect("Unexpected value type in formats"))
-                    .map(|a| Arc::new(a) as ArrayRef)
-                },
-                dt => return_compute_error_with!("timestamp_trunc does not support", dt),
-            )
+        dt => {
+            return_compute_error_with!("Unsupported input type for function 'timestamp_trunc'", dt)
         }
-        (dt, fmt) => Err(SparkError::Internal(format!(
-            "Unsupported datatype: {dt:}, format: {fmt:?} for function 'timestamp_trunc'"
-        ))),
-    }
+    };
+    let formats = match formats.data_type() {
+        DataType::Utf8 => formats.as_string::<i32>(),
+        fmt => return_compute_error_with!("timestamp_trunc does not support format type", fmt),
+    };
+    Ok(Arc::new(timestamp_trunc_by_row_format(array, formats)?))
 }
 
-macro_rules! timestamp_trunc_array_fmt_helper {
-    ($array: ident, $formats: ident, $datatype: ident) => {{
-        let mut builder = TimestampMicrosecondBuilder::with_capacity($array.len());
-        let iter = $array.into_iter();
-        assert_eq!(
-            $array.len(),
-            $formats.len(),
-            "lengths of values array and format array must be the same"
-        );
-        match $datatype {
-            DataType::Timestamp(TimeUnit::Microsecond, None) => {
-                // TimestampNTZ: operate directly on naive microsecond values
-                for (index, val) in iter.enumerate() {
-                    let micros_val = val.map(|v| i64::from(v));
-                    let trunc_fn = ntz_trunc_fn_for_format($formats.value(index))?;
-                    timestamp_trunc_ntz_single(micros_val, &mut builder, trunc_fn)?;
-                }
-                Ok(builder.finish())
-            }
-            DataType::Timestamp(TimeUnit::Microsecond, Some(tz_str)) => {
-                let tz: Tz = tz_str.parse()?;
-                for (index, val) in iter.enumerate() {
-                    let trunc_fn = tz_trunc_fn_for_format($formats.value(index))?;
-                    as_timestamp_tz_with_op_single::<T, _>(val, &mut builder, &tz, |dt| {
-                        as_micros_from_unix_epoch_utc(trunc_fn(dt))
-                    })?;
-                }
-                Ok(builder.finish().with_timezone(tz_str.as_ref()))
-            }
-            dt => {
-                return_compute_error_with!(
-                    "Unsupported input type '{:?}' for function 'timestamp_trunc'",
-                    dt
-                )
-            }
-        }
-    }};
+fn unpack_dictionary(array: &dyn Array) -> Result<ArrayRef, SparkError> {
+    match array.data_type() {
+        DataType::Dictionary(_, value_type) => arrow::compute::cast(array, value_type)
+            .map_err(|error| SparkError::Internal(error.to_string())),
+        _ => Ok(make_array(array.to_data())),
+    }
 }
 
-fn timestamp_trunc_array_fmt_plain_plain<T>(
-    array: &PrimitiveArray<T>,
+/// Truncates each row with the format in the same row.
+///
+/// Rows are grouped by format, and each group goes through the literal-format kernel with the
+/// other rows set to NULL. A row is therefore truncated by exactly the rules a literal format
+/// applies, including the DST handling, and a value in one group cannot make another group fail.
+/// A NULL format gives a NULL result, as in Spark.
+fn timestamp_trunc_by_row_format(
+    array: &TimestampMicrosecondArray,
     formats: &StringArray,
-) -> Result<TimestampMicrosecondArray, SparkError>
-where
-    T: ArrowTemporalType + ArrowNumericType,
-    i64: From<T::Native>,
-{
-    let data_type = array.data_type();
-    timestamp_trunc_array_fmt_helper!(array, formats, data_type)
-}
-fn timestamp_trunc_array_fmt_plain_dict<T, K>(
-    array: &PrimitiveArray<T>,
-    formats: &TypedDictionaryArray<K, StringArray>,
-) -> Result<TimestampMicrosecondArray, SparkError>
-where
-    T: ArrowTemporalType + ArrowNumericType,
-    i64: From<T::Native>,
-    K: ArrowDictionaryKeyType,
-{
-    let data_type = array.data_type();
-    timestamp_trunc_array_fmt_helper!(array, formats, data_type)
-}
+) -> Result<TimestampMicrosecondArray, SparkError> {
+    if array.len() != formats.len() {
+        return Err(SparkError::Internal(format!(
+            "timestamp_trunc has {} values but {} formats",
+            array.len(),
+            formats.len()
+        )));
+    }
 
-fn timestamp_trunc_array_fmt_dict_plain<T, K>(
-    array: &TypedDictionaryArray<K, PrimitiveArray<T>>,
-    formats: &StringArray,
-) -> Result<TimestampMicrosecondArray, SparkError>
-where
-    T: ArrowTemporalType + ArrowNumericType,
-    i64: From<T::Native>,
-    K: ArrowDictionaryKeyType,
-{
-    let data_type = array.values().data_type();
-    timestamp_trunc_array_fmt_helper!(array, formats, data_type)
-}
+    let mut granularities: Vec<&'static str> = Vec::new();
+    // A batch rarely holds more than a few spellings, so parse each one once.
+    let mut spellings: Vec<(&str, usize)> = Vec::new();
+    let mut row_groups: Vec<Option<usize>> = Vec::with_capacity(array.len());
+    let mut null_format_hides_value = false;
+    for index in 0..array.len() {
+        if array.is_null(index) {
+            row_groups.push(None);
+            continue;
+        }
+        if formats.is_null(index) {
+            null_format_hides_value = true;
+            row_groups.push(None);
+            continue;
+        }
+        let spelling = formats.value(index);
+        let group = match spellings.iter().find(|(seen, _)| *seen == spelling) {
+            Some((_, group)) => *group,
+            None => {
+                let granularity = normalize_timestamp_trunc_format(spelling)?;
+                let group = match granularities.iter().position(|g| *g == granularity) {
+                    Some(group) => group,
+                    None => {
+                        granularities.push(granularity);
+                        granularities.len() - 1
+                    }
+                };
+                spellings.push((spelling, group));
+                group
+            }
+        };
+        row_groups.push(Some(group));
+    }
 
-fn timestamp_trunc_array_fmt_dict_dict<T, K, F>(
-    array: &TypedDictionaryArray<K, PrimitiveArray<T>>,
-    formats: &TypedDictionaryArray<F, StringArray>,
-) -> Result<TimestampMicrosecondArray, SparkError>
-where
-    T: ArrowTemporalType + ArrowNumericType,
-    i64: From<T::Native>,
-    K: ArrowDictionaryKeyType,
-    F: ArrowDictionaryKeyType,
-{
-    let data_type = array.values().data_type();
-    timestamp_trunc_array_fmt_helper!(array, formats, data_type)
+    if granularities.len() == 1 && !null_format_hides_value {
+        return timestamp_trunc_upstream(array, granularities[0]);

Review Comment:
   Fixed in 6964f9237, after #6740 landed. The column-format path now passes `wrap_second_millisecond_overflow` to both calls into `timestamp_trunc_upstream`, so like the literal path it wraps before Spark 4.2 and raises `long overflow` from 4.2 on. `row_formats_follow_the_overflow_policy` covers a single-format column and a mixed one in both modes, and both of #6740's overflow fixtures now have a column-format case. On Spark 4.2 the column queries raise `long overflow`, and forcing the column path to wrap makes that fixture fail.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: truncate timestamps around DST transitions the way Spark does [datafusion-comet]

Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #6354:
URL: https://github.com/apache/datafusion-comet/pull/6354#discussion_r4146853670


##########
spark/src/test/resources/sql-tests/expressions/datetime/trunc_timestamp_dst_midnight.sql:
##########
@@ -0,0 +1,46 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements.  See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership.  The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License.  You may obtain a copy of the License at
+--
+--   http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied.  See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- DST transitions at midnight. America/Sao_Paulo skipped midnight on 2018-11-04,
+-- so DAY truncation and the day's start fall in the gap, and 23:00-24:00 on
+-- 2019-02-16 happened twice, once at -02:00 and once at -03:00.
+-- https://github.com/apache/datafusion-comet/issues/5633
+
+-- Config: spark.comet.expression.TruncTimestamp.allowIncompatible=true
+-- Config: spark.sql.session.timeZone=America/Sao_Paulo
+
+statement
+CREATE TABLE test_trunc_dst_midnight(ts timestamp) USING parquet
+
+statement
+INSERT INTO test_trunc_dst_midnight VALUES

Review Comment:
   Would it make sense to add a small `America/Havana` case where local midnight is ambiguous? Clocks fell back from 01:00 to 00:00 on 2020-11-01, and I don't think any current row makes `WEEK`, `MONTH`, `QUARTER` or `YEAR` land on a repeated local time, so the earlier-offset rule for those levels isn't pinned. I haven't run it, but for `timestamp('2020-11-01 00:30:00-05:00')` I'd expect `DAY` to give 00:00 at `-05:00` and `MONTH` to give 00:00 at `-04:00`. It would need its own session timezone, so probably a small new file.



##########
native/spark-expr/src/kernels/temporal.rs:
##########
@@ -101,80 +101,71 @@ fn trunc_days_to_week(days: i32) -> Option<i32> {
     Some(days - days_since_monday)
 }
 
-// Based on arrow_arith/temporal.rs:extract_component_from_datetime_array
-// Transforms an array of DateTime<Tz> to an array of TimestampMicrosecond after applying an
-// operation. The output array carries the input timezone annotation so downstream operators
-// (shuffle, sort, row converter) observe a matching schema.
-fn as_timestamp_tz_with_op<A: ArrayAccessor<Item = T::Native>, T: ArrowTemporalType, F>(
-    iter: ArrayIter<A>,
-    mut builder: PrimitiveBuilder<TimestampMicrosecondType>,
-    tz_str: &str,
-    op: F,
-) -> Result<TimestampMicrosecondArray, SparkError>
-where
-    F: Fn(DateTime<Tz>) -> i64,
-    i64: From<T::Native>,
-{
-    let tz: Tz = tz_str.parse()?;
-    for value in iter {
-        match value {
-            Some(value) => match as_datetime_with_timezone::<T>(value.into(), tz) {
-                Some(time) => builder.append_value(op(time)),
-                _ => {
-                    return Err(SparkError::Internal(
-                        "Unable to read value as datetime".to_string(),
-                    ));
-                }
-            },
-            None => builder.append_null(),
+/// How `date_trunc` truncates a timestamp with a timezone. Spark's `DateTimeUtils.truncTimestamp`
+/// treats the levels differently, and matching it matters around DST transitions.
+#[derive(Clone, Copy)]
+enum TzTrunc {
+    /// `MICROSECOND`, `MILLISECOND` and `SECOND`. Offsets are whole seconds, so Spark truncates the
+    /// instant itself. The value is the unit in microseconds.
+    Instant(i64),
+    /// `MINUTE`, `HOUR` and `DAY`. Spark uses `ZonedDateTime.truncatedTo`, which truncates the local
+    /// time and keeps the input's offset if the result is ambiguous.
+    LocalTime(NtzTruncFn),
+    /// `WEEK`, `MONTH`, `QUARTER` and `YEAR`. Spark truncates the local date and then takes
+    /// `LocalDate.atStartOfDay`, which uses the earlier offset if midnight is ambiguous.
+    LocalDate(NtzTruncFn),
+}
+
+/// Truncates `micros` in `tz` the way Spark's `DateTimeUtils.truncTimestamp` does. A truncated
+/// local time that falls in a DST gap takes the offset from before the gap, which gives the same
+/// instant as Java moving it forward by the gap's length. For the date levels that is also where
+/// `atStartOfDay` puts a day whose midnight falls in a gap that starts at midnight. Returns `None`
+/// if `micros` is out of chrono's range.
+fn trunc_timestamp_in_tz(micros: i64, tz: &Tz, trunc: TzTrunc) -> Option<i64> {
+    let (trunc_fn, keep_offset) = match trunc {
+        TzTrunc::Instant(unit) => return Some(micros - micros.rem_euclid(unit)),
+        TzTrunc::LocalTime(trunc_fn) => (trunc_fn, true),
+        TzTrunc::LocalDate(trunc_fn) => (trunc_fn, false),
+    };
+    let utc = DateTime::from_timestamp_micros(micros)?.naive_utc();

Review Comment:
   Would it make sense to reuse `micros_to_naive` and `naive_to_micros` here? This line and the last line of the function look like the same conversions, and the NTZ path in this file already uses them.



##########
native/spark-expr/src/kernels/temporal.rs:
##########
@@ -1238,6 +1225,52 @@ mod tests {
     /// pre-fix kernel reused the input's MST offset for the truncated date, producing a result
     /// one hour late. Also verifies the output array carries the input timezone, which is what
     /// allows the result to flow through shuffle/sort without a `RowConverter` schema mismatch.
+    /// Truncation around DST transitions, against java.time, which Spark's `truncTimestamp` uses.

Review Comment:
   Small thing: the new test seems to have landed between the existing `test_timestamp_trunc_dst_boundary` doc comment and its `#[test]` attribute. The Denver `QUARTER` lines above now read as part of this test's docs, and `test_timestamp_trunc_dst_boundary` has none. Would it make sense to move the new test above that comment block?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: truncate per-row date_trunc formats with the literal-format kernel [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #6354:
URL: https://github.com/apache/datafusion-comet/pull/6354#discussion_r4226319496


##########
native/spark-expr/src/datetime_funcs/timestamp_trunc.rs:
##########
@@ -133,7 +133,7 @@ impl PhysicalExpr for TimestampTruncExpr {
                 Ok(ColumnarValue::Array(relabel(result)?))
             }
             (ColumnarValue::Array(ts), ColumnarValue::Array(formats)) => {
-                let result = timestamp_trunc_array_fmt_dyn(&resolve_tz(ts)?, &formats)?;
+                let result = timestamp_trunc_array_fmt_dyn(&resolve_tz(ts)?, &formats, wrap)?;

Review Comment:
   This fixes a NULL format coming from a column. Two sibling shapes still fall through to the `_ =>` arm below, though, and fail with `Invalid input to function TimestampTrunc. Expected (Timestamp, Utf8)` under `allowIncompatible=true`:
   
   - a NULL literal format, such as `date_trunc(NULL, ts)`, where Spark returns NULL
   - a literal timestamp with a format column, such as `date_trunc(fmt, TIMESTAMP '2024-01-15 18:30:45')`
   
   I reproduced both by evaluating `TimestampTruncExpr` with `Literal(Utf8(None))` as the format and with a `TimestampMicrosecond` scalar as the child. Could you handle them here as well? A `Scalar(Utf8(None))` format can return a NULL array of the input's length. A scalar timestamp can be expanded with `to_array_of_size(formats.len())` before calling `timestamp_trunc_array_fmt_dyn`.
   



##########
spark/src/test/resources/sql-tests/expressions/datetime/trunc_timestamp_dst_format_column.sql:
##########
@@ -0,0 +1,66 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements.  See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership.  The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License.  You may obtain a copy of the License at
+--
+--   http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied.  See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+
+-- date_trunc with the format in a column, around DST overlaps, gaps and historical offsets. Each
+-- row is truncated with the rules a literal format applies (see trunc_timestamp_dst_ambiguous.sql
+-- for what each zone covers), and a NULL format gives NULL. A non-literal format is Incompatible,
+-- because an invalid format throws instead of returning NULL.
+
+-- Config: spark.comet.expression.TruncTimestamp.allowIncompatible=true
+-- Config: spark.sql.parquet.int96RebaseModeInWrite=CORRECTED
+-- ConfigMatrix: spark.sql.session.timeZone=UTC,America/Los_Angeles,America/New_York,America/Sao_Paulo,Africa/Monrovia,Asia/Aden,America/Havana,America/Toronto,America/Asuncion
+
+statement
+CREATE TABLE test_trunc_fmt_ts(ts timestamp) USING parquet
+
+statement
+INSERT INTO test_trunc_fmt_ts VALUES
+  (timestamp('1919-04-02T16:00:00Z')),
+  (timestamp('1919-03-31T04:30:00Z')),
+  (timestamp('1919-03-31T04:45:00Z')),
+  (timestamp('2023-10-15T12:00:00Z')),
+  (TIMESTAMP '1960-06-15 10:30:45'),
+  (timestamp('1947-03-13T20:53:30.123Z')),
+  (timestamp('1972-01-07T00:44:45Z')),
+  (timestamp('2018-11-04T02:30:15.123456Z')),
+  (timestamp('2018-11-04T03:30:15.123456Z')),
+  (timestamp('2024-03-10T09:30:15.123456Z')),
+  (timestamp('2024-03-10T10:30:15.123456Z')),
+  (timestamp('2024-11-03T08:30:15.123456Z')),
+  (timestamp('2024-11-03T09:30:15.123456Z')),
+  (timestamp('2020-11-01T04:30:00Z')),
+  (timestamp('2020-11-01T05:30:00Z')),
+  (timestamp('2020-11-15T12:00:00Z')),
+  (NULL)
+
+statement
+CREATE TABLE test_trunc_fmt_unit(fmt string) USING parquet
+
+statement
+INSERT INTO test_trunc_fmt_unit VALUES
+  ('YEAR'), ('yyyy'), ('YY'), ('QUARTER'), ('MONTH'), ('mon'), ('MM'), ('WEEK'), ('DAY'), ('dd'),
+  ('HOUR'), ('MINUTE'), ('SECOND'), ('MILLISECOND'), ('MICROSECOND'), (NULL)
+
+statement
+CREATE TABLE test_trunc_fmt(ts timestamp, fmt string) USING parquet
+
+statement
+INSERT INTO test_trunc_fmt SELECT ts, fmt FROM test_trunc_fmt_ts CROSS JOIN test_trunc_fmt_unit
+
+query expect_native(date_trunc)
+SELECT ts, fmt, date_trunc(fmt, ts) FROM test_trunc_fmt ORDER BY ts, fmt

Review Comment:
   Could you add a query for each of the two shapes from the `timestamp_trunc.rs` comment here? For example, `date_trunc(NULL, ts)` and `date_trunc(fmt, TIMESTAMP '2024-11-03 01:30:00')`, the second one in an overlap so it also exercises the DST rules. This file already runs with `allowIncompatible=true` across the nine zones, so it would pin both on the native path.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: truncate per-row date_trunc formats with the literal-format kernel [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #6354:
URL: https://github.com/apache/datafusion-comet/pull/6354#issuecomment-6083079684

   Thanks @viirya. The rest of your points are in 97728ba12 too:
   
   - The `timestamp_trunc_legacy` doc no longer mentions the row-format path, and it notes the far-future DST difference.
   - You're right about the fallback and Java's rules. I corrected the description and filed #6816 for far-future DST in native timezone conversions generally, since `hour` and the timestamp casts have the same gap.
   - I added `run-all-spark-profiles`.
   - `benches/timestamp_trunc.rs` has a `timestamp_trunc_format_column` group with a single format and mixed formats in UTC and `America/Los_Angeles`, on the same daytime instants I measured.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: truncate per-row date_trunc formats with the literal-format kernel [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #6354:
URL: https://github.com/apache/datafusion-comet/pull/6354#issuecomment-6086210294

   Thanks @viirya @parthchandra @sunchao @comphead for the reviews. Adding to merge queue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: truncate timestamps around DST transitions the way Spark does [datafusion-comet]

Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #6354:
URL: https://github.com/apache/datafusion-comet/pull/6354#issuecomment-5919287174

   A few review notes, comparing against Spark's `DateTimeUtils.truncTimestamp`:
   
   - **[correctness] `native/spark-expr/src/kernels/temporal.rs:143`** — the `LocalResult::None => resolve_local_datetime(...)` branch is still wrong for date-level truncation (WEEK/MONTH/QUARTER/YEAR) when a spring-forward gap straddles midnight. It shifts local time forward by the whole gap length, which is right for DAY/HOUR/MINUTE but not the date levels. Spark resolves those with `atStartOfDay`, which returns the gap's end instant, not midnight minus the pre-gap offset. They agree only when the gap begins exactly at midnight. Example: `America/Toronto`, `date_trunc('WEEK', ts)` on `1919-03-31 00:45:00-04:00` (gap ran 23:30 Mar 30 to 00:30 Mar 31). Spark returns `00:30:00-04:00`; this returns `01:00:00-04:00`, which is 30 minutes past the gap and actually later than the input instant. For the date levels, please resolve a gap to the transition boundary (the first valid instant at or after the truncated midnight) instead of the gap-length shift. (The bot flagged the same case.)
   
   - **[tests] `native/spark-expr/src/kernels/temporal.rs:1235`** — every DST case here (LA, Sao Paulo) has a gap that starts at midnight, so the date-level gap rule is never tested where it differs. Please add a straddles-midnight case (the Toronto 1919 one is clean) for WEEK/MONTH/QUARTER/YEAR, and an ambiguous-midnight zone like `America/Havana` (fall-back on 2020-11-01) to pin the earlier-offset rule for date-level overlaps.
   
   - **[tests] `native/spark-expr/src/kernels/temporal.rs:1236`** — `test_timestamp_trunc_matches_spark_at_dst_transitions` landed between the Denver doc comment and the `#[test]` it documents, so `test_timestamp_trunc_dst_boundary` lost its doc and the Denver comment now describes the wrong test. Move the new test above that comment block.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: truncate per-row date_trunc formats with the literal-format kernel [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #6354:
URL: https://github.com/apache/datafusion-comet/pull/6354#discussion_r4210197134


##########
native/spark-expr/src/kernels/temporal.rs:
##########
@@ -1260,154 +1213,109 @@ pub(crate) fn timestamp_trunc_array_fmt_dyn(
     array: &dyn Array,
     formats: &dyn Array,
 ) -> Result<ArrayRef, SparkError> {
-    match (array.data_type().clone(), formats.data_type().clone()) {
-        (DataType::Dictionary(_, _), DataType::Dictionary(_, _)) => {
-            downcast_dictionary_array!(
-                formats => {
-                    downcast_dictionary_array!(
-                        array => {
-                            timestamp_trunc_array_fmt_dict_dict(
-                                    &array.downcast_dict::<TimestampMicrosecondArray>().unwrap(),
-                                    &formats.downcast_dict::<StringArray>().unwrap())
-                            .map(|a| Arc::new(a) as ArrayRef)
-                        }
-                        dt => return_compute_error_with!("timestamp_trunc does not support", dt)
-                    )
-                }
-                fmt => return_compute_error_with!("timestamp_trunc does not support format type", fmt),
-            )
+    let array = unpack_dictionary(array)?;
+    let formats = unpack_dictionary(formats)?;
+    let array = match array.data_type() {
+        DataType::Timestamp(TimeUnit::Microsecond, _) => {
+            array.as_primitive::<TimestampMicrosecondType>()
         }
-        (DataType::Dictionary(_, _), DataType::Utf8) => {
-            downcast_dictionary_array!(
-                array => {
-                  timestamp_trunc_array_fmt_dict_plain(
-                        &array.downcast_dict::<PrimitiveArray<TimestampMicrosecondType>>().unwrap(),
-                        formats.as_any().downcast_ref::<StringArray>()
-                            .expect("Unexpected value type in formats"))
-                  .map(|a| Arc::new(a) as ArrayRef)
-                }
-                dt => return_compute_error_with!("timestamp_trunc does not support", dt),
-            )
-        }
-        (DataType::Timestamp(TimeUnit::Microsecond, _), DataType::Dictionary(_, _)) => {
-            downcast_dictionary_array!(
-                formats => {
-                downcast_temporal_array!(array => {
-                        timestamp_trunc_array_fmt_plain_dict(
-                                array,
-                                &formats.downcast_dict::<StringArray>().unwrap())
-                        .map(|a| Arc::new(a) as ArrayRef)
-                    }
-                    dt => return_compute_error_with!("timestamp_trunc does not support", dt),
-                    )
-                }
-                fmt => return_compute_error_with!("timestamp_trunc does not support format type", fmt),
-            )
-        }
-        (DataType::Timestamp(TimeUnit::Microsecond, _), DataType::Utf8) => {
-            downcast_temporal_array!(
-                array => {
-                    timestamp_trunc_array_fmt_plain_plain(array,
-                        formats.as_any().downcast_ref::<StringArray>().expect("Unexpected value type in formats"))
-                    .map(|a| Arc::new(a) as ArrayRef)
-                },
-                dt => return_compute_error_with!("timestamp_trunc does not support", dt),
-            )
+        dt => {
+            return_compute_error_with!("Unsupported input type for function 'timestamp_trunc'", dt)
         }
-        (dt, fmt) => Err(SparkError::Internal(format!(
-            "Unsupported datatype: {dt:}, format: {fmt:?} for function 'timestamp_trunc'"
-        ))),
-    }
+    };
+    let formats = match formats.data_type() {
+        DataType::Utf8 => formats.as_string::<i32>(),
+        fmt => return_compute_error_with!("timestamp_trunc does not support format type", fmt),
+    };
+    Ok(Arc::new(timestamp_trunc_by_row_format(array, formats)?))
 }
 
-macro_rules! timestamp_trunc_array_fmt_helper {
-    ($array: ident, $formats: ident, $datatype: ident) => {{
-        let mut builder = TimestampMicrosecondBuilder::with_capacity($array.len());
-        let iter = $array.into_iter();
-        assert_eq!(
-            $array.len(),
-            $formats.len(),
-            "lengths of values array and format array must be the same"
-        );
-        match $datatype {
-            DataType::Timestamp(TimeUnit::Microsecond, None) => {
-                // TimestampNTZ: operate directly on naive microsecond values
-                for (index, val) in iter.enumerate() {
-                    let micros_val = val.map(|v| i64::from(v));
-                    let trunc_fn = ntz_trunc_fn_for_format($formats.value(index))?;
-                    timestamp_trunc_ntz_single(micros_val, &mut builder, trunc_fn)?;
-                }
-                Ok(builder.finish())
-            }
-            DataType::Timestamp(TimeUnit::Microsecond, Some(tz_str)) => {
-                let tz: Tz = tz_str.parse()?;
-                for (index, val) in iter.enumerate() {
-                    let trunc_fn = tz_trunc_fn_for_format($formats.value(index))?;
-                    as_timestamp_tz_with_op_single::<T, _>(val, &mut builder, &tz, |dt| {
-                        as_micros_from_unix_epoch_utc(trunc_fn(dt))
-                    })?;
-                }
-                Ok(builder.finish().with_timezone(tz_str.as_ref()))
-            }
-            dt => {
-                return_compute_error_with!(
-                    "Unsupported input type '{:?}' for function 'timestamp_trunc'",
-                    dt
-                )
-            }
-        }
-    }};
+fn unpack_dictionary(array: &dyn Array) -> Result<ArrayRef, SparkError> {
+    match array.data_type() {
+        DataType::Dictionary(_, value_type) => arrow::compute::cast(array, value_type)
+            .map_err(|error| SparkError::Internal(error.to_string())),
+        _ => Ok(make_array(array.to_data())),
+    }
 }
 
-fn timestamp_trunc_array_fmt_plain_plain<T>(
-    array: &PrimitiveArray<T>,
+/// Truncates each row with the format in the same row.
+///
+/// Rows are grouped by format, and each group goes through the literal-format kernel with the
+/// other rows set to NULL. A row is therefore truncated by exactly the rules a literal format
+/// applies, including the DST handling, and a value in one group cannot make another group fail.
+/// A NULL format gives a NULL result, as in Spark.
+fn timestamp_trunc_by_row_format(
+    array: &TimestampMicrosecondArray,
     formats: &StringArray,
-) -> Result<TimestampMicrosecondArray, SparkError>
-where
-    T: ArrowTemporalType + ArrowNumericType,
-    i64: From<T::Native>,
-{
-    let data_type = array.data_type();
-    timestamp_trunc_array_fmt_helper!(array, formats, data_type)
-}
-fn timestamp_trunc_array_fmt_plain_dict<T, K>(
-    array: &PrimitiveArray<T>,
-    formats: &TypedDictionaryArray<K, StringArray>,
-) -> Result<TimestampMicrosecondArray, SparkError>
-where
-    T: ArrowTemporalType + ArrowNumericType,
-    i64: From<T::Native>,
-    K: ArrowDictionaryKeyType,
-{
-    let data_type = array.data_type();
-    timestamp_trunc_array_fmt_helper!(array, formats, data_type)
-}
+) -> Result<TimestampMicrosecondArray, SparkError> {
+    if array.len() != formats.len() {
+        return Err(SparkError::Internal(format!(
+            "timestamp_trunc has {} values but {} formats",
+            array.len(),
+            formats.len()
+        )));
+    }
 
-fn timestamp_trunc_array_fmt_dict_plain<T, K>(
-    array: &TypedDictionaryArray<K, PrimitiveArray<T>>,
-    formats: &StringArray,
-) -> Result<TimestampMicrosecondArray, SparkError>
-where
-    T: ArrowTemporalType + ArrowNumericType,
-    i64: From<T::Native>,
-    K: ArrowDictionaryKeyType,
-{
-    let data_type = array.values().data_type();
-    timestamp_trunc_array_fmt_helper!(array, formats, data_type)
-}
+    let mut granularities: Vec<&'static str> = Vec::new();
+    // A batch rarely holds more than a few spellings, so parse each one once.
+    let mut spellings: Vec<(&str, usize)> = Vec::new();
+    let mut row_groups: Vec<Option<usize>> = Vec::with_capacity(array.len());
+    let mut null_format_hides_value = false;
+    for index in 0..array.len() {
+        if array.is_null(index) {
+            row_groups.push(None);
+            continue;
+        }
+        if formats.is_null(index) {
+            null_format_hides_value = true;
+            row_groups.push(None);
+            continue;
+        }
+        let spelling = formats.value(index);
+        let group = match spellings.iter().find(|(seen, _)| *seen == spelling) {
+            Some((_, group)) => *group,
+            None => {
+                let granularity = normalize_timestamp_trunc_format(spelling)?;
+                let group = match granularities.iter().position(|g| *g == granularity) {
+                    Some(group) => group,
+                    None => {
+                        granularities.push(granularity);
+                        granularities.len() - 1
+                    }
+                };
+                spellings.push((spelling, group));
+                group
+            }
+        };
+        row_groups.push(Some(group));
+    }
 
-fn timestamp_trunc_array_fmt_dict_dict<T, K, F>(
-    array: &TypedDictionaryArray<K, PrimitiveArray<T>>,
-    formats: &TypedDictionaryArray<F, StringArray>,
-) -> Result<TimestampMicrosecondArray, SparkError>
-where
-    T: ArrowTemporalType + ArrowNumericType,
-    i64: From<T::Native>,
-    K: ArrowDictionaryKeyType,
-    F: ArrowDictionaryKeyType,
-{
-    let data_type = array.values().data_type();
-    timestamp_trunc_array_fmt_helper!(array, formats, data_type)
+    if granularities.len() == 1 && !null_format_hides_value {
+        return timestamp_trunc_upstream(array, granularities[0]);

Review Comment:
   [P2] Preserve Spark 4.2's checked overflow behavior when routing column formats here. With a UTC session, `spark.comet.expression.TruncTimestamp.allowIncompatible=true`, and columns containing `us=-9223372036854775808` and `fmt='SECOND'`, `unix_micros(date_trunc(fmt, timestamp_micros(us)))` now yields `9223372036854551616`. Spark 4.2 raises `ArithmeticException: long overflow`, and the base column path also errors. `MILLISECOND` similarly returns `9223372036854775616`. Although the scalar mismatch predates this PR, this new delegation materially worsens the column path by turning an error into a silently wrapped timestamp. Could you thread the version-aware overflow policy described for #6740 through both the single-group and masked-group calls, retaining wrapping for Spark 3.4–4.1, and add a Spark 4.2 column-format regression?
   
   Evidence: Compiled the unchanged base/head temporal kernels against locked Arrow 59.3.0, chrono 0.4.45 and DataFusion 55.1.0. For TimestampMicrosecondArray([i64::MIN]).with_timezone("UTC") and StringArray(["SECOND"]), the base returns Err("Unable to read value as datetime") and head returns 9223372036854551616. MILLISECOND returns 9223372036854775616 at head. Spark 4.2.0 DateTimeUtils.truncTimestamp and TruncTimestamp.eval with BoundReference column inputs both throw ArithmeticException: long overflow for both formats. Spark 4.2 source uses Math.subtractExact, while 3.4–4.1 use wrapping subtraction.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: truncate per-row date_trunc formats with the literal-format kernel [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #6354:
URL: https://github.com/apache/datafusion-comet/pull/6354


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org