You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "stevomitric (via GitHub)" <gi...@apache.org> on 2026/10/17 18:59:33 UTC
[PR] [SPARK-57826][SQL] Reject nanosecond timestamps in percentile_approx and histogram_numeric [spark]
stevomitric opened a new pull request, #59578:
URL: https://github.com/apache/spark/pull/59578
### What changes were proposed in this pull request?
`percentile_approx` / `approx_percentile` and `histogram_numeric` now reject `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)` (p in [7, 9]) with `DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE` in both ANSI modes, instead of implicitly casting them to `DATE`.
Neither function lists a nanosecond type in its input `TypeCollection`, so the implicit-cast fallback picks the first store-assignable member, which is `DateType`. For a nanosecond child, `inputTypes` now expects exactly that type, so the analyzer inserts no cast, and `checkInputDataTypes` reports the standard mismatch against the supported types. This is the same error the generic type coercion would give if it didn't pick `DATE`. Other input types, and the error messages for them, are unchanged.
Why reject instead of supporting nanos natively: both aggregates compute on doubles. `QuantileSummaries` keeps doubles as samples, and `NumericHistogram` averages double bin centers. A double has a 53-bit mantissa, so around 2020 any numeric encoding has a 256 ns resolution. That covers epoch nanos, epoch micros plus a fraction, and epoch seconds plus a fraction. Only 5 distinct doubles fall within one microsecond, and a single input `2020-01-01 12:00:00.123456789` would come back as `...123456768`. The sketch only compares values, so an order-preserving bit encoding could make it exact, but the nanos range (years 0001-9999) has about 2^68 values at p=9, which no 64-bit key can hold. Exact support would need a sketch with wider keys. Until then a clean error is better than a silently less precise result, and `CAST(c AS TIMESTAMP_NTZ)` / `CAST(c AS TIMESTAMP)` gives the microsecond result explicitly.
### Why are the changes needed?
Part of SPARK-56822. Today `SELECT percentile_approx(c, 0.5) FROM VALUES (TIMESTAMP_NTZ '2020-01-01 12:00:00.123456789') t(c)` returns `2020-01-01` typed `DATE` in both ANSI modes, and `histogram_numeric` returns `DATE` bin centers. This is a silent wrong result, which the SPIP's parity criterion rules out.
### Does this PR introduce _any_ user-facing change?
Yes, only for the nanosecond timestamp types, which are behind the preview flag `spark.sql.timestampNanosTypes.enabled`. These calls now fail at analysis instead of returning a `DATE`.
### How was this patch tested?
- New tests in `ApproximatePercentileSuite` and `HistogramNumericSuite`: NTZ/LTZ at p=7..9, both ANSI modes.
- New test in `ApproximatePercentileQuerySuite`: `percentile_approx` and `approx_percentile`, scalar and array percentages, both ANSI modes.
- New golden cases in `timestamp-ntz-nanos.sql` and `timestamp-ltz-nanos.sql`: both ANSI modes, plus the explicit-cast workaround.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Isaac
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60337][SQL] Reject nanosecond timestamps in percentile_approx and histogram_numeric [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b closed pull request #59578: [SPARK-60337][SQL] Reject nanosecond timestamps in percentile_approx and histogram_numeric
URL: https://github.com/apache/spark/pull/59578
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60337][SQL] Reject nanosecond timestamps in percentile_approx and histogram_numeric [spark]
Posted by "stevomitric (via GitHub)" <gi...@apache.org>.
stevomitric commented on PR #59578:
URL: https://github.com/apache/spark/pull/59578#issuecomment-6290459970
Thanks @uros-b for the review.
- JIRA: switched to SPARK-60337 and renamed the test references.
- Duplication: #59575 has merged and gives the same error, so I merged master and dropped the per-function logic and the duplicate golden cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60337][SQL] Reject nanosecond timestamps in percentile_approx and histogram_numeric [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #59578:
URL: https://github.com/apache/spark/pull/59578#issuecomment-6295252763
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/65680270d6be641823422e3c250cac19ea77ee36
- merged into branch-4.x https://github.com/apache/spark/commit/e2671665d0754244241ba64b1c191369bfcec118
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org