You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/11/08 06:26:26 UTC
[GitHub] [spark] panbingkun opened a new pull request, #38555: [SPARK-41044][SQL] Convert DATATYPE_MISMATCH.UNSPECIFIED_FRAME to INTERNAL_ERROR
panbingkun opened a new pull request, #38555:
URL: https://github.com/apache/spark/pull/38555
### What changes were proposed in this pull request?
The pr aims to convert DATATYPE_MISMATCH.UNSPECIFIED_FRAME to INTERNAL_ERROR.
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
1. Update existed UT.
2. Pass GA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] panbingkun commented on a diff in pull request #38555: [SPARK-41044][SQL] Convert DATATYPE_MISMATCH.UNSPECIFIED_FRAME to INTERNAL_ERROR
Posted by GitBox <gi...@apache.org>.
panbingkun commented on code in PR #38555:
URL: https://github.com/apache/spark/pull/38555#discussion_r1019878522
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala:
##########
@@ -57,16 +58,17 @@ case class WindowSpecDefinition(
frameSpecification = newChildren.last.asInstanceOf[WindowFrame])
override lazy val resolved: Boolean =
- childrenResolved && checkInputDataTypes().isSuccess &&
- frameSpecification.isInstanceOf[SpecifiedWindowFrame]
+ childrenResolved && frameSpecification.isInstanceOf[SpecifiedWindowFrame] &&
+ checkInputDataTypes().isSuccess
Review Comment:
boolean short circuit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #38555: [SPARK-41044][SQL] Convert DATATYPE_MISMATCH.UNSPECIFIED_FRAME to DATATYPE_MISMATCH.INTERNAL_ERROR
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #38555:
URL: https://github.com/apache/spark/pull/38555#discussion_r1016657369
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala:
##########
@@ -66,7 +66,13 @@ case class WindowSpecDefinition(
override def checkInputDataTypes(): TypeCheckResult = {
frameSpecification match {
case UnspecifiedFrame =>
- DataTypeMismatch(errorSubClass = "UNSPECIFIED_FRAME")
+ DataTypeMismatch(
+ errorSubClass = "INTERNAL_ERROR",
+ messageParameters = Map(
+ "message" -> ("Cannot use an UnspecifiedFrame. " +
+ "This should have been converted during analysis.")
Review Comment:
Not sure this is a good idea ... cc @MaxGekk @itholic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] panbingkun commented on a diff in pull request #38555: [SPARK-41044][SQL] Convert DATATYPE_MISMATCH.UNSPECIFIED_FRAME to INTERNAL_ERROR
Posted by GitBox <gi...@apache.org>.
panbingkun commented on code in PR #38555:
URL: https://github.com/apache/spark/pull/38555#discussion_r1019880123
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala:
##########
@@ -66,7 +66,13 @@ case class WindowSpecDefinition(
override def checkInputDataTypes(): TypeCheckResult = {
frameSpecification match {
case UnspecifiedFrame =>
- DataTypeMismatch(errorSubClass = "UNSPECIFIED_FRAME")
+ DataTypeMismatch(
+ errorSubClass = "INTERNAL_ERROR",
+ messageParameters = Map(
+ "message" -> ("Cannot use an UnspecifiedFrame. " +
+ "This should have been converted during analysis.")
Review Comment:
It is more reasonable to check `frameSpecification.isInstanceOf[SpecifiedWindowFrame]` before `checkInputDataTypes()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #38555: [SPARK-41044][SQL] Convert DATATYPE_MISMATCH.UNSPECIFIED_FRAME to INTERNAL_ERROR
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on PR #38555:
URL: https://github.com/apache/spark/pull/38555#issuecomment-1311446267
+1, LGTM. Merging to master.
Thank you, @panbingkun and @LuciferYang for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] panbingkun commented on a diff in pull request #38555: [WIP][SPARK-41044][SQL] Convert DATATYPE_MISMATCH.UNSPECIFIED_FRAME to DATATYPE_MISMATCH.INTERNAL_ERROR
Posted by GitBox <gi...@apache.org>.
panbingkun commented on code in PR #38555:
URL: https://github.com/apache/spark/pull/38555#discussion_r1016344333
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala:
##########
@@ -66,7 +66,13 @@ case class WindowSpecDefinition(
override def checkInputDataTypes(): TypeCheckResult = {
frameSpecification match {
case UnspecifiedFrame =>
- DataTypeMismatch(errorSubClass = "UNSPECIFIED_FRAME")
+ DataTypeMismatch(
+ errorSubClass = "INTERNAL_ERROR",
+ messageParameters = Map(
+ "message" -> ("Cannot use an UnspecifiedFrame. " +
+ "This should have been converted during analysis.")
Review Comment:
cannot throw an exception here, otherwise cannot continue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] panbingkun commented on a diff in pull request #38555: [SPARK-41044][SQL] Convert DATATYPE_MISMATCH.UNSPECIFIED_FRAME to INTERNAL_ERROR
Posted by GitBox <gi...@apache.org>.
panbingkun commented on code in PR #38555:
URL: https://github.com/apache/spark/pull/38555#discussion_r1019877660
##########
core/src/main/resources/error/error-classes.json:
##########
@@ -219,6 +219,11 @@
"Input to the function <functionName> cannot contain elements of the \"MAP\" type. In Spark, same maps may have different hashcode, thus hash expressions are prohibited on \"MAP\" elements. To restore previous behavior set \"spark.sql.legacy.allowHashOnMapType\" to \"true\"."
]
},
+ "INTERNAL_ERROR" : {
Review Comment:
Done
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala:
##########
@@ -66,7 +66,13 @@ case class WindowSpecDefinition(
override def checkInputDataTypes(): TypeCheckResult = {
frameSpecification match {
case UnspecifiedFrame =>
- DataTypeMismatch(errorSubClass = "UNSPECIFIED_FRAME")
+ DataTypeMismatch(
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #38555: [WIP][SPARK-41044][SQL] Convert DATATYPE_MISMATCH.UNSPECIFIED_FRAME to DATATYPE_MISMATCH.INTERNAL_ERROR
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on PR #38555:
URL: https://github.com/apache/spark/pull/38555#issuecomment-1306947596
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a diff in pull request #38555: [SPARK-41044][SQL] Convert DATATYPE_MISMATCH.UNSPECIFIED_FRAME to DATATYPE_MISMATCH.INTERNAL_ERROR
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on code in PR #38555:
URL: https://github.com/apache/spark/pull/38555#discussion_r1016704127
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala:
##########
@@ -66,7 +66,13 @@ case class WindowSpecDefinition(
override def checkInputDataTypes(): TypeCheckResult = {
frameSpecification match {
case UnspecifiedFrame =>
- DataTypeMismatch(errorSubClass = "UNSPECIFIED_FRAME")
+ DataTypeMismatch(
Review Comment:
@panbingkun Could you throw `SparkException.internalError`
##########
core/src/main/resources/error/error-classes.json:
##########
@@ -219,6 +219,11 @@
"Input to the function <functionName> cannot contain elements of the \"MAP\" type. In Spark, same maps may have different hashcode, thus hash expressions are prohibited on \"MAP\" elements. To restore previous behavior set \"spark.sql.legacy.allowHashOnMapType\" to \"true\"."
]
},
+ "INTERNAL_ERROR" : {
Review Comment:
I don't think we should introduce such sub-class. Could you re-use the existing `INTERNAL_ERROR`:
https://github.com/apache/spark/blob/31b923d50fa6176312f7217069c0055cd778788f/core/src/main/resources/error/error-classes.json#L557-L561
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala:
##########
@@ -66,7 +66,13 @@ case class WindowSpecDefinition(
override def checkInputDataTypes(): TypeCheckResult = {
frameSpecification match {
case UnspecifiedFrame =>
- DataTypeMismatch(errorSubClass = "UNSPECIFIED_FRAME")
+ DataTypeMismatch(
+ errorSubClass = "INTERNAL_ERROR",
+ messageParameters = Map(
+ "message" -> ("Cannot use an UnspecifiedFrame. " +
+ "This should have been converted during analysis.")
Review Comment:
Both checks exist in the source code from the first commit:
https://github.com/apache/spark/commit/f2c47082c3412a4cf8cbabe12585147c5ec3ea40
I would leaved `case UnspecifiedFrame` as is to cover the bad situation explicitly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] panbingkun commented on a diff in pull request #38555: [WIP][SPARK-41044][SQL] Convert DATATYPE_MISMATCH.UNSPECIFIED_FRAME to DATATYPE_MISMATCH.INTERNAL_ERROR
Posted by GitBox <gi...@apache.org>.
panbingkun commented on code in PR #38555:
URL: https://github.com/apache/spark/pull/38555#discussion_r1016493018
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala:
##########
@@ -66,7 +66,13 @@ case class WindowSpecDefinition(
override def checkInputDataTypes(): TypeCheckResult = {
frameSpecification match {
case UnspecifiedFrame =>
- DataTypeMismatch(errorSubClass = "UNSPECIFIED_FRAME")
+ DataTypeMismatch(
+ errorSubClass = "INTERNAL_ERROR",
+ messageParameters = Map(
+ "message" -> ("Cannot use an UnspecifiedFrame. " +
+ "This should have been converted during analysis.")
Review Comment:
https://github.com/apache/spark/blob/0eaa8e1e76ab6ecdd3b51d751857e50530ccdeb6/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala#L59-L61
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] panbingkun commented on a diff in pull request #38555: [SPARK-41044][SQL] Convert DATATYPE_MISMATCH.UNSPECIFIED_FRAME to INTERNAL_ERROR
Posted by GitBox <gi...@apache.org>.
panbingkun commented on code in PR #38555:
URL: https://github.com/apache/spark/pull/38555#discussion_r1019878119
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala:
##########
@@ -66,7 +66,13 @@ case class WindowSpecDefinition(
override def checkInputDataTypes(): TypeCheckResult = {
frameSpecification match {
case UnspecifiedFrame =>
- DataTypeMismatch(errorSubClass = "UNSPECIFIED_FRAME")
+ DataTypeMismatch(
+ errorSubClass = "INTERNAL_ERROR",
+ messageParameters = Map(
+ "message" -> ("Cannot use an UnspecifiedFrame. " +
+ "This should have been converted during analysis.")
Review Comment:
boolean short circuit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk closed pull request #38555: [SPARK-41044][SQL] Convert DATATYPE_MISMATCH.UNSPECIFIED_FRAME to INTERNAL_ERROR
Posted by GitBox <gi...@apache.org>.
MaxGekk closed pull request #38555: [SPARK-41044][SQL] Convert DATATYPE_MISMATCH.UNSPECIFIED_FRAME to INTERNAL_ERROR
URL: https://github.com/apache/spark/pull/38555
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org