You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "chenhao-db (via GitHub)" <gi...@apache.org> on 2026/10/16 19:45:53 UTC

[PR] [SPARK-60301] Use the session time zone when printing variant values as strings [spark]

chenhao-db opened a new pull request, #59561:
URL: https://github.com/apache/spark/pull/59561

   ### What changes were proposed in this pull request?
   
   Make variant-to-string conversion use the session time zone in all paths:
   
   - `ToStringBase`: add a `VariantType` case to the interpreted path and the codegen path. It calls `VariantVal.toJson(zoneId)`. Before, both paths fell through to `VariantVal.toString`, which always uses UTC. This fixes a variant nested in a struct, array, or map under `CAST(... AS STRING)`. It also fixes `ToPrettyString`, which `Dataset.show()` uses.
   - `HiveResult`: add the session `zoneId` to `TimeFormatters`, and use it for variants in `toHiveString`. This fixes the spark-sql CLI and Thrift server output.
   
   ### Why are the changes needed?
   
   A top-level `cast(v as string)` uses the cast's time zone, through `VariantGet.cast` and `VariantCastArgs`. A nested variant ignores that zone and always prints UTC. So the same value prints with a different offset at top level and when nested. `to_json` already uses the session time zone.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes. Variant timestamps now print in the session time zone, not in UTC, in these places:
   
   - a variant nested in a struct, array, or map that is cast to string
   - `Dataset.show()`
   - spark-sql CLI and Thrift server output
   
   The format does not change: it is still JSON with a `+HH:MM` offset. With `spark.sql.session.timeZone=America/Los_Angeles`:
   
   ```sql
   SELECT cast(named_struct('a', cast(TIMESTAMP'2020-01-01 00:00:00Z' AS VARIANT)) AS STRING);
   -- Before: {"2020-01-01 00:00:00+00:00"}
   -- After:  {"2019-12-31 16:00:00-08:00"}
   ```
   
   ### How was this patch tested?
   
   - New test in `VariantExpressionSuite`. It covers struct, array, and map casts and `ToPrettyString`, in UTC and in `America/Los_Angeles`, with codegen on and off.
   - New test in `HiveResultSuite`. It covers a top-level variant and a variant nested in an array.
   - Both new tests fail without the fix.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Opus 5.5


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-60301] Use the session time zone when printing variant values as strings [spark]

Posted by "cloud-fan (via GitHub) via reviews" <re...@spark.apache.org>.
cloud-fan commented on code in PR #59561:
URL: https://github.com/apache/spark/pull/59561#discussion_r4582226974


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ToStringBase.scala:
##########
@@ -100,6 +100,8 @@ trait ToStringBase { self: UnaryExpression with TimeZoneAwareExpression =>
       acceptAny[Long](t => UTF8String.fromString(timestampFormatter.format(t)))
     case TimestampNTZType =>
       acceptAny[Long](t => UTF8String.fromString(timestampNTZFormatter.format(t)))
+    case VariantType =>
+      acceptAny[VariantVal](v => UTF8String.fromString(v.toJson(zoneId)))

Review Comment:
   **Blocking (P1):** This arm makes the string form of a nested VARIANT depend on the cast's time zone, but `Cast.needsTimeZoneToString` (Cast.scala:456-461) still has no `VariantType` case. So `Cast.needsTimeZone` stays false for a cast to STRING of `ARRAY<VARIANT>`, `MAP<_, VARIANT>`, or `STRUCT<v: VARIANT>`. Its ScalaDoc (Cast.scala:451-454) still says a nested VARIANT renders at UTC, and the SPARK-60179 test in CastSuiteBase.scala (lines 2738-2746) still asserts that these casts do not need a time zone.
   
   Because of that, `Cast.canonicalized` drops the zone, so two casts that differ only in session time zone count as the same result for `sameResult`. Reading the code, `SELECT CAST(array(v) AS STRING) FROM t`, cached under UTC and run again after `SET spark.sql.session.timeZone=America/Los_Angeles`, is served from the cache as `["2020-01-01 00:00:00+00:00"]`, while the uncached query returns `["2019-12-31 16:00:00-08:00"]`. This is the pattern SPARK-60179 fixed for nested TIMESTAMP (CachedTableSuite.scala:2854-2882), and it also covers exchange reuse and sessions that share the cache manager. In addition, `Cast.resolved` accepts such a cast without a zone, and evaluating one now fails with `None.get` because this arm reads `zoneId`.
   
   The new tests cannot catch this: `checkEvaluation` assigns the session zone before evaluating, and the stale CastSuiteBase assertion keeps passing.
   
   **Recommended change:** Add a VariantType case returning true to Cast.needsTimeZoneToString in Cast.scala, and rewrite its ScalaDoc so it says that a VARIANT at any depth renders its TIMESTAMP values in the cast's time zone. In the Cast expression tests, move ARRAY<VARIANT>, MAP<INT, VARIANT>, and STRUCT<v: VARIANT> from the zone-independent list of the SPARK-60179 needsTimeZone test to its zone-dependent list and correct its comment, and give the complex-to-string resolution test a nested-variant case. In the cached-table tests, cover a column that casts an array containing a variant TIMESTAMP to string across session time zones.
   
   **Why this works:** With the new case, Cast.needsTimeZone(from, STRING) is true whenever the source contains a VARIANT. Cast.resolved then requires a zone, Cast.canonicalized keeps it, contextIndependentFoldable reports false, and the zone-gated builders attach the session zone. Two such casts in different zones then canonicalize differently, so QueryPlan.sameResult no longer matches them and the cache manager no longer serves text rendered in one zone to a query in another.
   
   **Compatibility:** The text that ToStringBase, ToPrettyString, and HiveResult produce for variants is unchanged, as is the classification of the top-level VARIANT cast and of every cast whose source contains no VARIANT.
   
   **Risks:** The classification is per type, so a cast to string of a struct, array, or map containing a VARIANT becomes zone-dependent even when its variants hold no TIMESTAMP value. Such plans are then no longer reused from a cache or exchange computed in another session time zone, and are no longer folded as context-independent constants, for example into a pushed-down Data Source V2 literal. Nested TIMESTAMP columns already work this way, and reverting the one case undoes it. Such casts also count as zone-dependent in the CHECK constraint write-time check and in Hive metastore partition pruning, which then handle them like other zone-dependent casts.
   
   **Constraints:** Leave every other case of Cast.needsTimeZone unchanged, including the top-level (VariantType, _) arm; only casts to string whose source contains a nested VARIANT change classification. Keep the new case unconditional even if a legacy setting is later added to restore UTC rendering, because declaring a zone-independent cast zone-dependent is safe while the reverse returns wrong results.
   
   **Exclusions:**
   - Whether the new rendering needs a migration-guide entry or a legacy setting is the separate decision about shipping the change ungated and is not part of this repair.
   
   **Verification:**
   - **Behavior:** Cast.needsTimeZone reports a cast to string as zone-dependent when its array, map, or struct source contains a VARIANT at any depth, and still reports the TIMESTAMP_NTZ, DATE, and TIME controls as zone-independent.
   - **Behavior:** A complex-to-string cast whose source contains a VARIANT is unresolved without a time zone, resolved with one, and not semantically equal when the only difference is UTC versus America/Los_Angeles.
   - **Regression:** Caching a query that casts an array containing a variant TIMESTAMP to string under UTC and rerunning it under America/Los_Angeles returns the -08:00 rendering instead of the cached +00:00 text.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-60301][SQL] Use the session time zone when printing variant values as strings [spark]

Posted by "cloud-fan (via GitHub) via reviews" <re...@spark.apache.org>.
cloud-fan commented on PR #59561:
URL: https://github.com/apache/spark/pull/59561#issuecomment-6996841394

   **Merge Summary:**
   - merged into master https://github.com/apache/spark/commit/d157ec95b9b63977c114f6027e9400ac15dad82c
   - merged into branch-4.x https://github.com/apache/spark/commit/8f09b40052406f672d193937f05e828df473c913
   - merged into branch-4.4 https://github.com/apache/spark/commit/733d6c51d00937da019734dd8259d74f5371a006
   
   *Posted by `merge_spark_pr.py`*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-60301][SQL] Use the session time zone when printing variant values as strings [spark]

Posted by "cloud-fan (via GitHub) via reviews" <re...@spark.apache.org>.
cloud-fan closed pull request #59561: [SPARK-60301][SQL] Use the session time zone when printing variant values as strings
URL: https://github.com/apache/spark/pull/59561


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-60301] Use the session time zone when printing variant values as strings [spark]

Posted by "chenhao-db (via GitHub)" <gi...@apache.org>.
chenhao-db commented on PR #59561:
URL: https://github.com/apache/spark/pull/59561#issuecomment-6224575122

   @cloud-fan could you take a look? Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SQL][SPARK-60301] Use the session time zone when printing variant values as strings [spark]

Posted by "chenhao-db (via GitHub) via reviews" <re...@spark.apache.org>.
chenhao-db commented on code in PR #59561:
URL: https://github.com/apache/spark/pull/59561#discussion_r4590354233


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ToStringBase.scala:
##########
@@ -100,6 +100,8 @@ trait ToStringBase { self: UnaryExpression with TimeZoneAwareExpression =>
       acceptAny[Long](t => UTF8String.fromString(timestampFormatter.format(t)))
     case TimestampNTZType =>
       acceptAny[Long](t => UTF8String.fromString(timestampNTZFormatter.format(t)))
+    case VariantType =>
+      acceptAny[VariantVal](v => UTF8String.fromString(v.toJson(zoneId)))

Review Comment:
   Thanks. I address the review comments, please take another look.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org