You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "stevomitric (via GitHub)" <gi...@apache.org> on 2026/07/13 13:20:52 UTC
[PR] [WIP][SPARK-57811][SQL] Support string to nanosecond-precision timestamp coercion in comparisons and predicates [spark]
stevomitric opened a new pull request, #57223:
URL: https://github.com/apache/spark/pull/57223
### What changes were proposed in this pull request?
Add non-ANSI type-coercion arms so that a string compared against a nanosecond-precision timestamp column (`TIMESTAMP_NTZ(p)`/`TIMESTAMP_LTZ(p)`, p in [7, 9]) is cast to that nanos type, mirroring the existing microsecond `TimestampType` handling (`StringPromotionTypeCoercion` equality arms + `TypeCoercion.findCommonTypeForBinaryComparison`).
### Why are the changes needed?
Without a nanos arm, a string compared to a nanos column loses sub-microsecond precision under legacy `spark.sql.legacy.typeCoercion.datetimeToString.enabled=true` instead of parsing at full precision like the microsecond types. ANSI mode is unchanged.
### Does this PR introduce _any_ user-facing change?
Yes. String vs nanosecond-timestamp comparisons now coerce the string to the nanos type with full precision (preview `spark.sql.timestampNanosTypes.enabled`).
### How was this patch tested?
Extended existing suits and added golden file tests.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57811][SQL] Support string to nanosecond-precision timestamp coercion in comparisons and predicates [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #57223:
URL: https://github.com/apache/spark/pull/57223#issuecomment-5128260005
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/54dce691f533cb558a7c8403e90d90b8c971dc50
- merged into branch-4.x https://github.com/apache/spark/commit/dc31dc41f3afe7243a7ab1966bb77c505f5de3aa
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57811][SQL] Support string to nanosecond-precision timestamp coercion in comparisons and predicates [spark]
Posted by "stevomitric (via GitHub)" <gi...@apache.org>.
stevomitric commented on code in PR #57223:
URL: https://github.com/apache/spark/pull/57223#discussion_r3673857006
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionSuite.scala:
##########
@@ -1701,6 +1701,66 @@ class TypeCoercionSuite extends TypeCoercionSuiteBase {
EqualTo(Cast(date0301, TimestampType), timestamp0301000000))
ruleTest(rule, LessThan(date0301, timestamp0301000001),
LessThan(Cast(date0301, TimestampType), timestamp0301000001))
+
Review Comment:
added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57811][SQL] Support string to nanosecond-precision timestamp coercion in comparisons and predicates [spark]
Posted by "stevomitric (via GitHub)" <gi...@apache.org>.
stevomitric commented on PR #57223:
URL: https://github.com/apache/spark/pull/57223#issuecomment-4994035095
> can you fix merge conflicts?
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP][SPARK-57811][SQL] Support string to nanosecond-precision timestamp coercion in comparisons and predicates [spark]
Posted by "stevomitric (via GitHub)" <gi...@apache.org>.
stevomitric commented on PR #57223:
URL: https://github.com/apache/spark/pull/57223#issuecomment-4992164746
cc @cloud-fan, @MaxGekk PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57811][SQL] Support string to nanosecond-precision timestamp coercion in comparisons and predicates [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #57223:
URL: https://github.com/apache/spark/pull/57223#discussion_r3645271755
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala:
##########
@@ -145,6 +145,14 @@ object TypeCoercion extends TypeCoercionBase {
=> if (conf.castDatetimeToString) Some(st) else Some(TimestampType)
case (TimestampType, st: StringType)
=> if (conf.castDatetimeToString) Some(st) else Some(TimestampType)
+ // Mirror the micros TimestampType (LTZ) arms above: nanos TIMESTAMP_LTZ(p) honors
+ // castDatetimeToString. Nanos TIMESTAMP_NTZ(p) intentionally has no arm here and, exactly like
+ // micros TimestampNTZType, falls through to the config-blind canPromoteAsInBinaryComparison
+ // line below -- so the LTZ and NTZ families stay consistent across micros and nanos.
+ case (st: StringType, tsNanos: TimestampLTZNanosType)
+ => if (conf.castDatetimeToString) Some(st) else Some(tsNanos)
+ case (tsNanos: TimestampLTZNanosType, st: StringType)
+ => if (conf.castDatetimeToString) Some(st) else Some(tsNanos)
Review Comment:
Please check - the new range arms make nanos TIMESTAMP_NTZ(p) honor castDatetimeToString, but micros TimestampNTZType has no arm here and stays config-blind (falls through to canPromoteAsInBinaryComparison, verified in master). So under legacy castDatetimeToString=true, non-ANSI range comparison, string < ts_ntz_micros stays a timestamp comparison while string < ts_ntz_nanos(9) becomes lexicographic string comparison. Is this intentional?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57811][SQL] Support string to nanosecond-precision timestamp coercion in comparisons and predicates [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #57223:
URL: https://github.com/apache/spark/pull/57223#discussion_r3645274103
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionSuite.scala:
##########
@@ -1701,6 +1701,66 @@ class TypeCoercionSuite extends TypeCoercionSuiteBase {
EqualTo(Cast(date0301, TimestampType), timestamp0301000000))
ruleTest(rule, LessThan(date0301, timestamp0301000001),
LessThan(Cast(date0301, TimestampType), timestamp0301000001))
+
Review Comment:
How about `AnsiTypeCoercionSuite`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57811][SQL] Support string to nanosecond-precision timestamp coercion in comparisons and predicates [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #57223:
URL: https://github.com/apache/spark/pull/57223#issuecomment-5069826836
Also, please resolve conflicts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57811][SQL] Support string to nanosecond-precision timestamp coercion in comparisons and predicates [spark]
Posted by "stevomitric (via GitHub)" <gi...@apache.org>.
stevomitric commented on code in PR #57223:
URL: https://github.com/apache/spark/pull/57223#discussion_r3673873411
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala:
##########
@@ -145,6 +145,14 @@ object TypeCoercion extends TypeCoercionBase {
=> if (conf.castDatetimeToString) Some(st) else Some(TimestampType)
case (TimestampType, st: StringType)
=> if (conf.castDatetimeToString) Some(st) else Some(TimestampType)
+ // Mirror the micros TimestampType (LTZ) arms above: nanos TIMESTAMP_LTZ(p) honors
+ // castDatetimeToString. Nanos TIMESTAMP_NTZ(p) intentionally has no arm here and, exactly like
+ // micros TimestampNTZType, falls through to the config-blind canPromoteAsInBinaryComparison
+ // line below -- so the LTZ and NTZ families stay consistent across micros and nanos.
+ case (st: StringType, tsNanos: TimestampLTZNanosType)
+ => if (conf.castDatetimeToString) Some(st) else Some(tsNanos)
+ case (tsNanos: TimestampLTZNanosType, st: StringType)
+ => if (conf.castDatetimeToString) Some(st) else Some(tsNanos)
Review Comment:
NTZ doesn't depend on the config here, the new arm is only for TimestampLTZNanosType, (no NTZ arm), so `string < ts_ntz_nanos` falls through and stays a timestamp comparison, same as micros TimestampNTZType.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57811][SQL] Support string to nanosecond-precision timestamp coercion in comparisons and predicates [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #57223:
URL: https://github.com/apache/spark/pull/57223#issuecomment-4992793734
can you fix merge conflicts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57811][SQL] Support string to nanosecond-precision timestamp coercion in comparisons and predicates [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b closed pull request #57223: [SPARK-57811][SQL] Support string to nanosecond-precision timestamp coercion in comparisons and predicates
URL: https://github.com/apache/spark/pull/57223
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57811][SQL] Support string to nanosecond-precision timestamp coercion in comparisons and predicates [spark]
Posted by "stevomitric (via GitHub)" <gi...@apache.org>.
stevomitric commented on PR #57223:
URL: https://github.com/apache/spark/pull/57223#issuecomment-5063687792
cc @uros-b PTAL when you get a chance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org