You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "wangyum (via GitHub)" <gi...@apache.org> on 2023/03/31 07:00:28 UTC
[GitHub] [spark] wangyum opened a new pull request, #40616: [SPARK-42991][SQL] Disable string type +/- interval in ANSI mode
wangyum opened a new pull request, #40616:
URL: https://github.com/apache/spark/pull/40616
### What changes were proposed in this pull request?
1. Moved `ResolveBinaryArithmetic` from `Analyzer` to `TypeCoercion` and `AnsiTypeCoercion`.
2. Update `ResolveBinaryArithmetic` in `AnsiTypeCoercion` to only support datetime types +/- intervals.
### Why are the changes needed?
Avoid potential data issues.
### Does this PR introduce _any_ user-facing change?
Disable string +/- interval in ANSI mode.
### How was this patch tested?
Update existing test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #40616: [SPARK-42991][SQL] Disable string type +/- interval in ANSI mode
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #40616:
URL: https://github.com/apache/spark/pull/40616#issuecomment-1494792083
@wangyum Do you mind writing more details about the reason why making this change in the PR description?
On second thought: as I will propose enabling ANSI mode by default in Spark 4.0, this one adds more gaps between the default behavior and ANSI SQL mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #40616: [SPARK-42991][SQL] Disable string type +/- interval in ANSI mode
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on PR #40616:
URL: https://github.com/apache/spark/pull/40616#issuecomment-1491411368
cc @cloud-fan @gengliangwang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #40616: [SPARK-42991][SQL] Disable string type +/- interval in ANSI mode
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40616:
URL: https://github.com/apache/spark/pull/40616#issuecomment-1657307919
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a diff in pull request #40616: [SPARK-42991][SQL] Disable string type +/- interval in ANSI mode
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #40616:
URL: https://github.com/apache/spark/pull/40616#discussion_r1155515408
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/AnsiTypeCoercion.scala:
##########
@@ -288,6 +290,82 @@ object AnsiTypeCoercion extends TypeCoercionBase {
}
}
+ // Please see the comments in `TypeCoercion.ResolveBinaryArithmetic`.
+ object ResolveBinaryArithmetic extends TypeCoercionRule {
Review Comment:
Yes. Updated the code:
https://github.com/apache/spark/blob/cd21cce9131e17d53b164fdfe8d54a59a265f41a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala#L1173-L1194
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a diff in pull request #40616: [SPARK-42991][SQL] Disable string type +/- interval in ANSI mode
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #40616:
URL: https://github.com/apache/spark/pull/40616#discussion_r1156585817
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala:
##########
@@ -1171,6 +1172,29 @@ object TypeCoercion extends TypeCoercionBase {
}
}
+ // For legacy support. For example: string type +/- interval.
+ object ResolveBinaryArithmetic extends TypeCoercionRule {
+ override val transform: PartialFunction[Expression, Expression] =
+ AnsiTypeCoercion.ResolveBinaryArithmetic.transform.andThen {
+ case a @ Add(l, r, _) if a.childrenResolved => (l.dataType, r.dataType) match {
+ case (StringType | _: NumericType | BinaryType | BooleanType,
+ CalendarIntervalType | _: DayTimeIntervalType) =>
+ Cast(TimeAdd(l, r), l.dataType)
Review Comment:
For ANSI mode. It should be [DatetimeType and AnsiIntervalType](https://github.com/apache/spark/pull/40616/files#diff-2b33eb8ae2127bd8a3d461132f4db0c57c65419aebe2f4a14d595b6ea5cebf36R341-R343):
For non-ANSI mode. It should be DatetimeType, AnsiIntervalType, StringType, NumericType, BinaryType and BooleanType.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #40616: [SPARK-42991][SQL] Disable string type +/- interval in ANSI mode
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on PR #40616:
URL: https://github.com/apache/spark/pull/40616#issuecomment-1517869092
@gengliangwang Has updated the description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a diff in pull request #40616: [SPARK-42991][SQL] Disable string type +/- interval in ANSI mode
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #40616:
URL: https://github.com/apache/spark/pull/40616#discussion_r1156313326
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala:
##########
@@ -1171,6 +1172,29 @@ object TypeCoercion extends TypeCoercionBase {
}
}
+ // For legacy support. For example: string type +/- interval.
+ object ResolveBinaryArithmetic extends TypeCoercionRule {
+ override val transform: PartialFunction[Expression, Expression] =
+ AnsiTypeCoercion.ResolveBinaryArithmetic.transform.andThen {
+ case a @ Add(l, r, _) if a.childrenResolved => (l.dataType, r.dataType) match {
+ case (StringType | _: NumericType | BinaryType | BooleanType,
+ CalendarIntervalType | _: DayTimeIntervalType) =>
+ Cast(TimeAdd(l, r), l.dataType)
Review Comment:
The first child of `TimeAdd` has to be `AnyTimestampType` but it can be `StringType | _: NumericType | BinaryType | BooleanType` here..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #40616: [SPARK-42991][SQL] Disable string type +/- interval in ANSI mode
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #40616: [SPARK-42991][SQL] Disable string type +/- interval in ANSI mode
URL: https://github.com/apache/spark/pull/40616
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #40616: [SPARK-42991][SQL] Disable string type +/- interval in ANSI mode
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #40616:
URL: https://github.com/apache/spark/pull/40616#discussion_r1154114177
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/AnsiTypeCoercion.scala:
##########
@@ -288,6 +290,82 @@ object AnsiTypeCoercion extends TypeCoercionBase {
}
}
+ // Please see the comments in `TypeCoercion.ResolveBinaryArithmetic`.
+ object ResolveBinaryArithmetic extends TypeCoercionRule {
Review Comment:
is it possible to do some code sharing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org