You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/04/21 08:19:51 UTC
[GitHub] [spark] panbingkun opened a new pull request, #36298: [SPARK-38740][SQL][TESTS] Test the error class: INVALID_JSON_SCHEMA_MAPTYPE
panbingkun opened a new pull request, #36298:
URL: https://github.com/apache/spark/pull/36298
## What changes were proposed in this pull request?
This PR aims to add a test for the error class INVALID_JSON_SCHEMA_MAPTYPE to `QueryCompilationErrorsSuite`.
### Why are the changes needed?
The changes improve test coverage, and document expected error messages in tests.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
By running new test:
```
$ build/sbt "sql/testOnly *QueryCompilationErrorsSuite*"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] panbingkun commented on pull request #36298: [SPARK-38740][SQL][TESTS] Test the error class: INVALID_JSON_SCHEMA_MAPTYPE
Posted by GitBox <gi...@apache.org>.
panbingkun commented on PR #36298:
URL: https://github.com/apache/spark/pull/36298#issuecomment-1107415203
> @panbingkun Could you resolve conflicts, please.
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a diff in pull request #36298: [SPARK-38740][SQL][TESTS] Test the error class: INVALID_JSON_SCHEMA_MAPTYPE
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on code in PR #36298:
URL: https://github.com/apache/spark/pull/36298#discussion_r856090883
##########
sql/core/src/test/scala/org/apache/spark/sql/errors/QueryCompilationErrorsSuite.scala:
##########
@@ -380,6 +380,24 @@ class QueryCompilationErrorsSuite
}
}
}
+
+ test("INVALID_JSON_SCHEMA_MAPTYPE: " +
+ "Parse JSON rows can only contain StringType as a key type for a MapType.") {
+ withTempPath { path =>
+ val schema = StructType(
+ StructField("map", MapType(IntegerType, IntegerType, true), false) :: Nil)
+
+ checkErrorClass(
+ exception = intercept[AnalysisException] {
+ spark.read.schema(schema).json(path.getCanonicalPath)
Review Comment:
I think no need to create a temporary dir. AFAK, `json()` accepts a `Dataset`. Could you pass an empty dataset to `json()`, please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a diff in pull request #36298: [SPARK-38740][SQL][TESTS] Test the error class: INVALID_JSON_SCHEMA_MAPTYPE
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on code in PR #36298:
URL: https://github.com/apache/spark/pull/36298#discussion_r855790900
##########
sql/core/src/test/scala/org/apache/spark/sql/errors/QueryCompilationErrorsSuite.scala:
##########
@@ -364,6 +364,21 @@ class QueryCompilationErrorsSuite extends QueryTest with SharedSparkSession {
}
}
}
+
+ test("INVALID_JSON_SCHEMA_MAPTYPE: " +
+ "Parse JSON rows can only contain StringType as a key type for a MapType.") {
+ withTempPath { path =>
+ val schema = StructType(
+ StructField("map", MapType(IntegerType, IntegerType, true), false) :: Nil)
+ val e = intercept[AnalysisException] {
+ spark.read.schema(schema).json(path.getCanonicalPath)
+ }
+ assert(e.getErrorClass === "INVALID_JSON_SCHEMA_MAPTYPE")
+ assert(e.getMessage ===
+ "Input schema StructType(StructField(map,MapType(IntegerType,IntegerType,true),false)) " +
+ "can only contain StringType as a key type for a MapType.")
Review Comment:
Could you use the `checkErrorClass` method added by https://github.com/apache/spark/pull/36287.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #36298: [SPARK-38740][SQL][TESTS] Test the error class: INVALID_JSON_SCHEMA_MAPTYPE
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on PR #36298:
URL: https://github.com/apache/spark/pull/36298#issuecomment-1107389402
@panbingkun Could you resolve conflicts, please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #36298: [SPARK-38740][SQL][TESTS] Test the error class: INVALID_JSON_SCHEMA_MAPTYPE
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on PR #36298:
URL: https://github.com/apache/spark/pull/36298#issuecomment-1107780461
+1, LGTM. Merging to master.
Thank you, @panbingkun.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk closed pull request #36298: [SPARK-38740][SQL][TESTS] Test the error class: INVALID_JSON_SCHEMA_MAPTYPE
Posted by GitBox <gi...@apache.org>.
MaxGekk closed pull request #36298: [SPARK-38740][SQL][TESTS] Test the error class: INVALID_JSON_SCHEMA_MAPTYPE
URL: https://github.com/apache/spark/pull/36298
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #36298: [SPARK-38740][SQL][TESTS] Test the error class: INVALID_JSON_SCHEMA_MAPTYPE
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on PR #36298:
URL: https://github.com/apache/spark/pull/36298#issuecomment-1106603604
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org