You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/10/30 12:35:16 UTC
[GitHub] [spark] panbingkun opened a new pull request, #38438: [SPARK-40748][SQL] Migrate type check failures of conditions onto error classes
panbingkun opened a new pull request, #38438:
URL: https://github.com/apache/spark/pull/38438
### What changes were proposed in this pull request?
This pr replaces TypeCheckFailure by DataTypeMismatch in type checks in the misc expressions, includes:
1. If (2): https://github.com/apache/spark/blob/1431975723d8df30a25b2333eddcfd0bb6c57677/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala#L61-L67
2. CaseWhen (2): https://github.com/apache/spark/blob/1431975723d8df30a25b2333eddcfd0bb6c57677/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala#L175-L183
3. InSubquery (2):
https://github.com/apache/spark/blob/1431975723d8df30a25b2333eddcfd0bb6c57677/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala#L378-L396
4. In (1):
https://github.com/apache/spark/blob/1431975723d8df30a25b2333eddcfd0bb6c57677/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala#L453
### Why are the changes needed?
Migration onto error classes unifies Spark SQL error messages.
### Does this PR introduce _any_ user-facing change?
Yes. The PR changes user-facing error messages.
### How was this patch tested?
1. Add new UT
2. Update existed UT
3. Pass GA
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a diff in pull request #38438: [SPARK-40748][SQL] Migrate type check failures of conditions onto error classes
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on code in PR #38438:
URL: https://github.com/apache/spark/pull/38438#discussion_r1010145931
##########
sql/core/src/test/java/test/org/apache/spark/sql/JavaColumnExpressionSuite.java:
##########
@@ -79,12 +80,8 @@ public void isInCollectionCheckExceptionMessage() {
createStructField("a", IntegerType, false),
createStructField("b", createArrayType(IntegerType, false), false)));
Dataset<Row> df = spark.createDataFrame(rows, schema);
- Exception e = Assert.assertThrows(Exception.class,
+ AnalysisException e = Assert.assertThrows(AnalysisException.class,
() -> df.filter(df.col("a").isInCollection(Arrays.asList(new Column("b")))));
- Arrays.asList("cannot resolve",
- "due to data type mismatch: Arguments must be same type but were")
- .forEach(s ->
- Assert.assertTrue(e.getMessage().toLowerCase(Locale.ROOT)
- .contains(s.toLowerCase(Locale.ROOT))));
+ Assert.assertTrue(e.message().startsWith("[DATATYPE_MISMATCH.DATA_DIFF_TYPES]"));
Review Comment:
e is an `AnalysisException`, correct? Can't you just check its `errorClass`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk closed pull request #38438: [SPARK-40748][SQL] Migrate type check failures of conditions onto error classes
Posted by GitBox <gi...@apache.org>.
MaxGekk closed pull request #38438: [SPARK-40748][SQL] Migrate type check failures of conditions onto error classes
URL: https://github.com/apache/spark/pull/38438
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a diff in pull request #38438: [SPARK-40748][SQL] Migrate type check failures of conditions onto error classes
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on code in PR #38438:
URL: https://github.com/apache/spark/pull/38438#discussion_r1009486429
##########
sql/core/src/test/java/test/org/apache/spark/sql/JavaColumnExpressionSuite.java:
##########
@@ -81,10 +81,13 @@ public void isInCollectionCheckExceptionMessage() {
Dataset<Row> df = spark.createDataFrame(rows, schema);
Exception e = Assert.assertThrows(Exception.class,
() -> df.filter(df.col("a").isInCollection(Arrays.asList(new Column("b")))));
- Arrays.asList("cannot resolve",
- "due to data type mismatch: Arguments must be same type but were")
- .forEach(s ->
- Assert.assertTrue(e.getMessage().toLowerCase(Locale.ROOT)
- .contains(s.toLowerCase(Locale.ROOT))));
+ System.out.println(e.getMessage().toLowerCase(Locale.ROOT));
Review Comment:
Is it needed?
##########
sql/core/src/test/java/test/org/apache/spark/sql/JavaColumnExpressionSuite.java:
##########
@@ -81,10 +81,13 @@ public void isInCollectionCheckExceptionMessage() {
Dataset<Row> df = spark.createDataFrame(rows, schema);
Exception e = Assert.assertThrows(Exception.class,
() -> df.filter(df.col("a").isInCollection(Arrays.asList(new Column("b")))));
- Arrays.asList("cannot resolve",
- "due to data type mismatch: Arguments must be same type but were")
- .forEach(s ->
- Assert.assertTrue(e.getMessage().toLowerCase(Locale.ROOT)
- .contains(s.toLowerCase(Locale.ROOT))));
+ System.out.println(e.getMessage().toLowerCase(Locale.ROOT));
+ Arrays.asList(
+ "datatype_mismatch.data_diff_types",
+ "cannot resolve \"(a in (b))\"",
+ "due to data type mismatch: input to `in` should all be the same type, " +
+ "but it's [\"int\", \"array<int>\"].").forEach(s ->
+ Assert.assertTrue(e.getMessage().toLowerCase(Locale.ROOT)
+ .contains(s.toLowerCase(Locale.ROOT))));
Review Comment:
Let's don't compare error messages. If you cannot call `checkError()` easily, just check error classes (maybe more if you can).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #38438: [SPARK-40748][SQL] Migrate type check failures of conditions onto error classes
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on PR #38438:
URL: https://github.com/apache/spark/pull/38438#issuecomment-1299581375
+1, LGTM. Merging to master.
Thank you, @panbingkun.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] panbingkun commented on a diff in pull request #38438: [SPARK-40748][SQL] Migrate type check failures of conditions onto error classes
Posted by GitBox <gi...@apache.org>.
panbingkun commented on code in PR #38438:
URL: https://github.com/apache/spark/pull/38438#discussion_r1010013950
##########
sql/core/src/test/java/test/org/apache/spark/sql/JavaColumnExpressionSuite.java:
##########
@@ -81,10 +81,13 @@ public void isInCollectionCheckExceptionMessage() {
Dataset<Row> df = spark.createDataFrame(rows, schema);
Exception e = Assert.assertThrows(Exception.class,
() -> df.filter(df.col("a").isInCollection(Arrays.asList(new Column("b")))));
- Arrays.asList("cannot resolve",
- "due to data type mismatch: Arguments must be same type but were")
- .forEach(s ->
- Assert.assertTrue(e.getMessage().toLowerCase(Locale.ROOT)
- .contains(s.toLowerCase(Locale.ROOT))));
+ System.out.println(e.getMessage().toLowerCase(Locale.ROOT));
+ Arrays.asList(
+ "datatype_mismatch.data_diff_types",
+ "cannot resolve \"(a in (b))\"",
+ "due to data type mismatch: input to `in` should all be the same type, " +
+ "but it's [\"int\", \"array<int>\"].").forEach(s ->
+ Assert.assertTrue(e.getMessage().toLowerCase(Locale.ROOT)
+ .contains(s.toLowerCase(Locale.ROOT))));
Review Comment:
I will check:
1.Exception Type -> AnalysisException
2.Error classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] panbingkun commented on a diff in pull request #38438: [SPARK-40748][SQL] Migrate type check failures of conditions onto error classes
Posted by GitBox <gi...@apache.org>.
panbingkun commented on code in PR #38438:
URL: https://github.com/apache/spark/pull/38438#discussion_r1010349649
##########
sql/core/src/test/java/test/org/apache/spark/sql/JavaColumnExpressionSuite.java:
##########
@@ -79,12 +80,8 @@ public void isInCollectionCheckExceptionMessage() {
createStructField("a", IntegerType, false),
createStructField("b", createArrayType(IntegerType, false), false)));
Dataset<Row> df = spark.createDataFrame(rows, schema);
- Exception e = Assert.assertThrows(Exception.class,
+ AnalysisException e = Assert.assertThrows(AnalysisException.class,
() -> df.filter(df.col("a").isInCollection(Arrays.asList(new Column("b")))));
- Arrays.asList("cannot resolve",
- "due to data type mismatch: Arguments must be same type but were")
- .forEach(s ->
- Assert.assertTrue(e.getMessage().toLowerCase(Locale.ROOT)
- .contains(s.toLowerCase(Locale.ROOT))));
+ Assert.assertTrue(e.message().startsWith("[DATATYPE_MISMATCH.DATA_DIFF_TYPES]"));
Review Comment:
e is an AnalysisException
<img width="877" alt="image" src="https://user-images.githubusercontent.com/15246973/199227396-98442bc1-f5d6-4779-8e09-b21691b7ade5.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] panbingkun commented on a diff in pull request #38438: [SPARK-40748][SQL] Migrate type check failures of conditions onto error classes
Posted by GitBox <gi...@apache.org>.
panbingkun commented on code in PR #38438:
URL: https://github.com/apache/spark/pull/38438#discussion_r1010013499
##########
sql/core/src/test/java/test/org/apache/spark/sql/JavaColumnExpressionSuite.java:
##########
@@ -81,10 +81,13 @@ public void isInCollectionCheckExceptionMessage() {
Dataset<Row> df = spark.createDataFrame(rows, schema);
Exception e = Assert.assertThrows(Exception.class,
() -> df.filter(df.col("a").isInCollection(Arrays.asList(new Column("b")))));
- Arrays.asList("cannot resolve",
- "due to data type mismatch: Arguments must be same type but were")
- .forEach(s ->
- Assert.assertTrue(e.getMessage().toLowerCase(Locale.ROOT)
- .contains(s.toLowerCase(Locale.ROOT))));
+ System.out.println(e.getMessage().toLowerCase(Locale.ROOT));
Review Comment:
Sorry, I left the debugging code by mistake.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #38438: [SPARK-40748][SQL] Migrate type check failures of conditions onto error classes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on PR #38438:
URL: https://github.com/apache/spark/pull/38438#issuecomment-1296354425
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a diff in pull request #38438: [SPARK-40748][SQL] Migrate type check failures of conditions onto error classes
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on code in PR #38438:
URL: https://github.com/apache/spark/pull/38438#discussion_r1010683264
##########
sql/core/src/test/java/test/org/apache/spark/sql/JavaColumnExpressionSuite.java:
##########
@@ -79,12 +83,16 @@ public void isInCollectionCheckExceptionMessage() {
createStructField("a", IntegerType, false),
createStructField("b", createArrayType(IntegerType, false), false)));
Dataset<Row> df = spark.createDataFrame(rows, schema);
- Exception e = Assert.assertThrows(Exception.class,
+ AnalysisException e = Assert.assertThrows(AnalysisException.class,
() -> df.filter(df.col("a").isInCollection(Arrays.asList(new Column("b")))));
- Arrays.asList("cannot resolve",
- "due to data type mismatch: Arguments must be same type but were")
- .forEach(s ->
- Assert.assertTrue(e.getMessage().toLowerCase(Locale.ROOT)
- .contains(s.toLowerCase(Locale.ROOT))));
+ Assert.assertTrue(e.getErrorClass().equals("DATATYPE_MISMATCH.DATA_DIFF_TYPES"));
+ Map<String, String> messageParameters = new HashMap() {
+ {
Review Comment:
Just in case, are the inner `{}` really needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #38438: [SPARK-40748][SQL] Migrate type check failures of conditions onto error classes
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on PR #38438:
URL: https://github.com/apache/spark/pull/38438#issuecomment-1296765409
@panbingkun Could you resolve conflicts, please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org