You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/10/21 11:49:40 UTC
[GitHub] [spark] panbingkun opened a new pull request, #38332: [SPARK-40750][SQL] Migrate type check failures of math expressions onto error classes
panbingkun opened a new pull request, #38332:
URL: https://github.com/apache/spark/pull/38332
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
- Add new UT.
- Update Existed UT.
- Pass GA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #38332: [SPARK-40750][SQL] Migrate type check failures of math expressions onto error classes
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on PR #38332:
URL: https://github.com/apache/spark/pull/38332#issuecomment-1289273157
+1, LGTM. Merging to master.
Thank you, @panbingkun.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a diff in pull request #38332: [SPARK-40750][SQL] Migrate type check failures of math expressions onto error classes
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on code in PR #38332:
URL: https://github.com/apache/spark/pull/38332#discussion_r1002667108
##########
core/src/main/resources/error/error-classes.json:
##########
@@ -143,6 +143,11 @@
"Offset expression <offset> must be a literal."
]
},
+ "INVALID_ELEMENT_TYPE" : {
+ "message" : [
+ "Input to function <functionName> cannot contain elements of <elementType>. <message>"
Review Comment:
Let's don't pass `message`, and introduce more specific error class which includes the message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a diff in pull request #38332: [SPARK-40750][SQL] Migrate type check failures of math expressions onto error classes
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on code in PR #38332:
URL: https://github.com/apache/spark/pull/38332#discussion_r1002891939
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala:
##########
@@ -268,15 +269,17 @@ abstract class HashExpression[E] extends Expression {
override def checkInputDataTypes(): TypeCheckResult = {
if (children.length < 1) {
- TypeCheckResult.TypeCheckFailure(
- s"input to function $prettyName requires at least one argument")
+ DataTypeMismatch(
+ errorSubClass = "WRONG_NUM_PARAMS",
+ messageParameters = Map(
+ "functionName" -> prettyName,
+ "expectedNum" -> "> 0",
+ "actualNum" -> children.length.toString))
} else if (children.exists(child => hasMapType(child.dataType)) &&
!SQLConf.get.getConf(SQLConf.LEGACY_ALLOW_HASH_ON_MAPTYPE)) {
- TypeCheckResult.TypeCheckFailure(
- s"input to function $prettyName cannot contain elements of MapType. In Spark, same maps " +
- "may have different hashcode, thus hash expressions are prohibited on MapType elements." +
- s" To restore previous behavior set ${SQLConf.LEGACY_ALLOW_HASH_ON_MAPTYPE.key} " +
- "to true.")
+ DataTypeMismatch(
+ errorSubClass = "HASH_MAP_TYPE",
+ messageParameters = Map("functionName" -> prettyName))
Review Comment:
Quote by `toSQLId()`
##########
core/src/main/resources/error/error-classes.json:
##########
@@ -143,6 +143,11 @@
"Offset expression <offset> must be a literal."
]
},
+ "HASH_MAP_TYPE" : {
+ "message" : [
+ "Input to function <functionName> cannot contain elements of MAP. In Spark, same maps may have different hashcode, thus hash expressions are prohibited on MapType elements. To restore previous behavior set \"spark.sql.legacy.allowHashOnMapType\" to \"true\".",
Review Comment:
```suggestion
"Input to the function <functionName> cannot contain elements of the \"MAP\" type. In Spark, same maps may have different hashcode, thus hash expressions are prohibited on \"MAP\" elements. To restore previous behavior set \"spark.sql.legacy.allowHashOnMapType\" to \"true\".",
```
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala:
##########
@@ -268,15 +269,17 @@ abstract class HashExpression[E] extends Expression {
override def checkInputDataTypes(): TypeCheckResult = {
if (children.length < 1) {
- TypeCheckResult.TypeCheckFailure(
- s"input to function $prettyName requires at least one argument")
+ DataTypeMismatch(
+ errorSubClass = "WRONG_NUM_PARAMS",
+ messageParameters = Map(
+ "functionName" -> prettyName,
Review Comment:
Please, quote the function id by `toSQLId()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] panbingkun commented on a diff in pull request #38332: [SPARK-40750][SQL] Migrate type check failures of math expressions onto error classes
Posted by GitBox <gi...@apache.org>.
panbingkun commented on code in PR #38332:
URL: https://github.com/apache/spark/pull/38332#discussion_r1002824150
##########
core/src/main/resources/error/error-classes.json:
##########
@@ -143,6 +143,11 @@
"Offset expression <offset> must be a literal."
]
},
+ "INVALID_ELEMENT_TYPE" : {
+ "message" : [
+ "Input to function <functionName> cannot contain elements of <elementType>. <message>"
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #38332: [WIP][SPARK-40750][SQL] Migrate type check failures of math expressions onto error classes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on PR #38332:
URL: https://github.com/apache/spark/pull/38332#issuecomment-1287088983
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] panbingkun commented on a diff in pull request #38332: [SPARK-40750][SQL] Migrate type check failures of math expressions onto error classes
Posted by GitBox <gi...@apache.org>.
panbingkun commented on code in PR #38332:
URL: https://github.com/apache/spark/pull/38332#discussion_r1002904666
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala:
##########
@@ -268,15 +269,17 @@ abstract class HashExpression[E] extends Expression {
override def checkInputDataTypes(): TypeCheckResult = {
if (children.length < 1) {
- TypeCheckResult.TypeCheckFailure(
- s"input to function $prettyName requires at least one argument")
+ DataTypeMismatch(
+ errorSubClass = "WRONG_NUM_PARAMS",
+ messageParameters = Map(
+ "functionName" -> prettyName,
+ "expectedNum" -> "> 0",
+ "actualNum" -> children.length.toString))
} else if (children.exists(child => hasMapType(child.dataType)) &&
!SQLConf.get.getConf(SQLConf.LEGACY_ALLOW_HASH_ON_MAPTYPE)) {
- TypeCheckResult.TypeCheckFailure(
- s"input to function $prettyName cannot contain elements of MapType. In Spark, same maps " +
- "may have different hashcode, thus hash expressions are prohibited on MapType elements." +
- s" To restore previous behavior set ${SQLConf.LEGACY_ALLOW_HASH_ON_MAPTYPE.key} " +
- "to true.")
+ DataTypeMismatch(
+ errorSubClass = "HASH_MAP_TYPE",
+ messageParameters = Map("functionName" -> prettyName))
Review Comment:
Done
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala:
##########
@@ -268,15 +269,17 @@ abstract class HashExpression[E] extends Expression {
override def checkInputDataTypes(): TypeCheckResult = {
if (children.length < 1) {
- TypeCheckResult.TypeCheckFailure(
- s"input to function $prettyName requires at least one argument")
+ DataTypeMismatch(
+ errorSubClass = "WRONG_NUM_PARAMS",
+ messageParameters = Map(
+ "functionName" -> prettyName,
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] panbingkun commented on a diff in pull request #38332: [SPARK-40750][SQL] Migrate type check failures of math expressions onto error classes
Posted by GitBox <gi...@apache.org>.
panbingkun commented on code in PR #38332:
URL: https://github.com/apache/spark/pull/38332#discussion_r1002898122
##########
core/src/main/resources/error/error-classes.json:
##########
@@ -143,6 +143,11 @@
"Offset expression <offset> must be a literal."
]
},
+ "HASH_MAP_TYPE" : {
+ "message" : [
+ "Input to function <functionName> cannot contain elements of MAP. In Spark, same maps may have different hashcode, thus hash expressions are prohibited on MapType elements. To restore previous behavior set \"spark.sql.legacy.allowHashOnMapType\" to \"true\".",
Review Comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk closed pull request #38332: [SPARK-40750][SQL] Migrate type check failures of math expressions onto error classes
Posted by GitBox <gi...@apache.org>.
MaxGekk closed pull request #38332: [SPARK-40750][SQL] Migrate type check failures of math expressions onto error classes
URL: https://github.com/apache/spark/pull/38332
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org