You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "stevomitric (via GitHub)" <gi...@apache.org> on 2026/07/29 17:23:23 UTC
[PR] [WIP][SPARK-57841][SQL][TEST] Add cross-precision and nested-type test cov… [spark]
stevomitric opened a new pull request, #57634:
URL: https://github.com/apache/spark/pull/57634
### What changes were proposed in this pull request?
This is a test-only change under the nanosecond-timestamp SPIP (SPARK-56822). It adds end-to-end coverage for operations over the nanosecond-precision timestamp types `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)` (`p` in `[7, 9]`) that ride on the cross-precision widening (SPARK-57454) and string/nanos coercion (SPARK-57811) but had no dedicated tests:
- Map-key lookup fast path: nanos-key sections added to the threshold-parameterized `GetMapValue` (`ComplexTypeSuite`) and `ElementAt` (`CollectionExpressionsSuite`) blocks, exercising both the hash (threshold 0) and linear (threshold Int.MaxValue) paths in interpreted and codegen modes.
- Mixed-precision joins: `TimestampNanosJoinSuiteBase` gains `nanos(p1) JOIN nanos(p2)` and `micro JOIN nanos` cases across every forced join strategy and codegen mode, asserting the join key widens to the common nanos type and still distinguishes the sub-microsecond remainder.
- New `TimestampNanosRenderingSuiteBase` for `show()` string rendering and `collect()` external round-trip at `p = 7/8/9`.
- Golden SQL sections appended to `timestamp-{ntz,ltz}-nanos.sql` for INTERSECT/EXCEPT, BETWEEN, if/nvl/ifnull, IN (subquery), explode, element_at, struct-field extraction, and map lookup by string key and by nanosecond key.
The map-key hash fast path was flagged as a possible latent issue (its hash arm lists only microsecond timestamps). It is in fact already correct: nanos keys are physical `TimestampNanosVal` objects that hash and compare through the object `hashCode()`/ `equals()` fall-through arm, consistently on the driver and codegen sides. The new tests lock this in; no production change was required.
### Why are the changes needed?
To establish and guard the behavior of nanosecond-precision timestamps across joins, set operations, conditionals, subqueries, complex-type access, and rendering.
### Does this PR introduce any user-facing change?
No. Test-only.
### How was this patch tested?
New and updated tests:
- `catalyst/testOnly *.ComplexTypeSuite *.CollectionExpressionsSuite`
- `sql/testOnly *.TimestampNanosJoin*Suite *.TimestampNanosRendering*Suite`
- `sql/testOnly *.SQLQueryTestSuite -- -z nanos`
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57841][SQL][TEST] Add cross-precision and nested-type test coverage for nanosecond-precision timestamps [spark]
Posted by "stevomitric (via GitHub)" <gi...@apache.org>.
stevomitric commented on PR #57634:
URL: https://github.com/apache/spark/pull/57634#issuecomment-5152353122
> oh there are merge conflicts
should be resolved now. cc @uros-b , @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57841][SQL][TEST] Add cross-precision and nested-type test coverage for nanosecond-precision timestamps [spark]
Posted by "stevomitric (via GitHub)" <gi...@apache.org>.
stevomitric commented on code in PR #57634:
URL: https://github.com/apache/spark/pull/57634#discussion_r3689472554
##########
sql/core/src/test/scala/org/apache/spark/sql/TimestampNanosRenderingSuiteBase.scala:
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import java.time.{Instant, LocalDateTime}
+
+import org.apache.spark.SparkConf
+// castToImpl: `spark.createDataFrame` is typed as the public sql.DataFrame, but `showString` is a
+// classic-only method; this implicit narrows the receiver to classic.Dataset for that call.
+import org.apache.spark.sql.classic.ClassicConversions.castToImpl
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+
+/**
+ * End-to-end `show()` / `collect()` rendering tests over the nanosecond-precision timestamp types
+ * `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)` (`p` in `[7, 9]`). These ride on the nanos cast-to-string
+ * (`ToPrettyString` -> `ToStringBase`, which routes nanos through the Types Framework fraction
+ * formatter) and the nanos external encoders, so no production change is required.
+ *
+ * Two distinct surfaces are pinned, neither of which the golden SQL files can express:
+ * - `Dataset.show()` string rendering: the fractional second is rendered to the type's precision
+ * with sub-precision digits floored to zero and an all-zero fraction trimmed entirely. A value
+ * that only differs below the type's precision therefore renders identically -- the display is
+ * lossy at `p`, exactly as the stored value is. (The golden `.out` files render via
+ * `hiveResultString`, a different entry point onto the same formatter; `HiveResultSuite` covers
+ * that one. `show()` is what a user actually sees at the console.)
+ * - `collect()` external round-trip: NTZ comes back as `java.time.LocalDateTime` and LTZ as
+ * `java.time.Instant` (see `RowEncoder`), and `.getNano` preserves the sub-microsecond
+ * remainder floored to the type's precision -- so two values that share `epochMicros` but
+ * differ in `nanosWithinMicro` are distinguishable after a full driver round-trip.
+ *
+ * The nanosecond timestamp types are gated behind a preview flag enabled by default under tests
+ * (`Utils.isTesting`), so it is not set here. The session time zone is fixed to America/Los_Angeles
+ * (UTC-08:00, no DST on 2020-01-01) so the `TIMESTAMP_LTZ` (`Instant`) values render
+ * deterministically in wall-clock time. The two subclasses run every test with ANSI mode on/off.
+ */
+abstract class TimestampNanosRenderingSuiteBase extends SharedSparkSession {
Review Comment:
sure thing. accepted your change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57841][SQL][TEST] Add cross-precision and nested-type test coverage for nanosecond-precision timestamps [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #57634:
URL: https://github.com/apache/spark/pull/57634#issuecomment-5159505143
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/920c92890192fe3831c02ddb6fcf321caa053f06
- merged into branch-4.x https://github.com/apache/spark/commit/6bac1dbd7cf7ad0ba5852f5ad40263918e22cbda
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57841][SQL][TEST] Add cross-precision and nested-type test coverage for nanosecond-precision timestamps [spark]
Posted by "stevomitric (via GitHub)" <gi...@apache.org>.
stevomitric commented on PR #57634:
URL: https://github.com/apache/spark/pull/57634#issuecomment-5132115052
cc @uros-b , @cloud-fan PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57841][SQL][TEST] Add cross-precision and nested-type test coverage for nanosecond-precision timestamps [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b closed pull request #57634: [SPARK-57841][SQL][TEST] Add cross-precision and nested-type test coverage for nanosecond-precision timestamps
URL: https://github.com/apache/spark/pull/57634
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57841][SQL][TEST] Add cross-precision and nested-type test coverage for nanosecond-precision timestamps [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #57634:
URL: https://github.com/apache/spark/pull/57634#discussion_r3684791290
##########
sql/core/src/test/scala/org/apache/spark/sql/TimestampNanosRenderingSuiteBase.scala:
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import java.time.{Instant, LocalDateTime}
+
+import org.apache.spark.SparkConf
+// castToImpl: `spark.createDataFrame` is typed as the public sql.DataFrame, but `showString` is a
+// classic-only method; this implicit narrows the receiver to classic.Dataset for that call.
+import org.apache.spark.sql.classic.ClassicConversions.castToImpl
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+
+/**
+ * End-to-end `show()` / `collect()` rendering tests over the nanosecond-precision timestamp types
+ * `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)` (`p` in `[7, 9]`). These ride on the nanos cast-to-string
+ * (`ToPrettyString` -> `ToStringBase`, which routes nanos through the Types Framework fraction
+ * formatter) and the nanos external encoders, so no production change is required.
+ *
+ * Two distinct surfaces are pinned, neither of which the golden SQL files can express:
+ * - `Dataset.show()` string rendering: the fractional second is rendered to the type's precision
+ * with sub-precision digits floored to zero and an all-zero fraction trimmed entirely. A value
+ * that only differs below the type's precision therefore renders identically -- the display is
+ * lossy at `p`, exactly as the stored value is. (The golden `.out` files render via
+ * `hiveResultString`, a different entry point onto the same formatter; `HiveResultSuite` covers
+ * that one. `show()` is what a user actually sees at the console.)
+ * - `collect()` external round-trip: NTZ comes back as `java.time.LocalDateTime` and LTZ as
+ * `java.time.Instant` (see `RowEncoder`), and `.getNano` preserves the sub-microsecond
+ * remainder floored to the type's precision -- so two values that share `epochMicros` but
+ * differ in `nanosWithinMicro` are distinguishable after a full driver round-trip.
+ *
+ * The nanosecond timestamp types are gated behind a preview flag enabled by default under tests
+ * (`Utils.isTesting`), so it is not set here. The session time zone is fixed to America/Los_Angeles
+ * (UTC-08:00, no DST on 2020-01-01) so the `TIMESTAMP_LTZ` (`Instant`) values render
+ * deterministically in wall-clock time. The two subclasses run every test with ANSI mode on/off.
+ */
+abstract class TimestampNanosRenderingSuiteBase extends SharedSparkSession {
Review Comment:
Please mix in `QueryTest` explicitly, as required for new SQL/DataFrame suites; relying on `SharedSparkSession`'s historical inheritance makes this suite depend on an implementation detail.
```suggestion
abstract class TimestampNanosRenderingSuiteBase extends QueryTest with SharedSparkSession {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57841][SQL][TEST] Add cross-precision and nested-type test coverage for nanosecond-precision timestamps [spark]
Posted by "stevomitric (via GitHub)" <gi...@apache.org>.
stevomitric commented on PR #57634:
URL: https://github.com/apache/spark/pull/57634#issuecomment-5141481246
Thanks @cloud-fan and @uros-b for the review! Addressed comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57841][SQL][TEST] Add cross-precision and nested-type test coverage for nanosecond-precision timestamps [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #57634:
URL: https://github.com/apache/spark/pull/57634#issuecomment-5143102824
oh there are merge conflicts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org