[jira] [Updated] (SPARK-57826) Support nanosecond-precision timestamps in approx_percentile/percentile_approx and histogram_numeric

["Max Gekk (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-57826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Max Gekk updated SPARK-57826:
-----------------------------
    Shepherd: Max Gekk

> Support nanosecond-precision timestamps in approx_percentile/percentile_approx and histogram_numeric
> ----------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-57826
>                 URL: https://issues.apache.org/jira/browse/SPARK-57826
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>    Affects Versions: 4.3.0
>            Reporter: Max Gekk
>            Priority: Major
>
> This sub-task is part of the umbrella SPARK-56822 (timestamps with nanosecond precision).
> h2. Problem
> {{ApproximatePercentile}} (aggregate/ApproximatePercentile.scala ~L104-209) lists {{TimestampType}}, {{TimestampNTZType}}, {{AnyTimeType}} in {{inputTypes}} but omits {{AnyTimestampNanoType}}; its value path does {{value.toDouble}} / result {{.toLong}}, which is wrong for {{TimestampNanosVal}} (not a {{Number}}). {{HistogramNumeric}} (aggregate/HistogramNumeric.scala ~L80-168) has the same pattern ({{asInstanceOf[Number].doubleValue()}}). Microsecond timestamps work; nanosecond types are rejected at analysis and would also fail at runtime. This mirrors the TIME extension added by SPARK-57557.
> h2. Goal
> Accept {{AnyTimestampNanoType}} in both aggregates and convert {{TimestampNanosVal}} to/from the internal double representation without losing sub-microsecond precision (e.g. via epoch seconds + fractional nanos), returning a nanosecond timestamp at the input precision/family.
> h2. Scope
> Extend {{inputTypes}}; add nanosecond value<->double conversion in update/merge/eval; preserve precision on the result type. Follow the SPARK-57557 pattern.
> h2. Acceptance criteria
> * {{approx_percentile}} / {{percentile_approx}} / {{histogram_numeric}} over NTZ/LTZ nanosecond timestamps return correctly-typed nanosecond results; accuracy comparable to the microsecond path.
> h2. Testing
> {{ApproximatePercentileQuerySuite}}, {{HistogramNumericSuite}}; nanos golden files.
> h2. Dependencies
> None - independent (mirrors the TIME work SPARK-57557).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org