You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2023/01/11 23:21:29 UTC
[GitHub] [spark] eric-maynard opened a new pull request, #39519: [SPARK-41995] Accept non-foldable expressions in schema_of_json
eric-maynard opened a new pull request, #39519:
URL: https://github.com/apache/spark/pull/39519
### What changes were proposed in this pull request?
Presently, only foldable expressions can be passed in to schema_of_json, e.g. `SCHEMA_OF_JSON(CONCAT('', ''))`.
With this change, we accept _any_ string expression and infer the JSON schema if possible.
### Why are the changes needed?
Users may have a column containing JSON strings, but today they can't find the schema of each record. With this change, finding the schema of a given column is possible.
### Does this PR introduce _any_ user-facing change?
Yes, schema_of_json will now accept string expressions which are not foldable.
### How was this patch tested?
New tests were added in `JsonFunctionsSuite`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ggwiebe commented on pull request #39519: [SPARK-41995][SQL] Accept non-foldable expressions in schema_of_json
Posted by "ggwiebe (via GitHub)" <gi...@apache.org>.
ggwiebe commented on PR #39519:
URL: https://github.com/apache/spark/pull/39519#issuecomment-1398786764
This is great!
I have a customer that I am helping where they are receiving many different types of messages enveloped into a structured stream. Today I cannot give a schema for the message because there are many different message types stored in this column (each row could hold a different message type, with a different schema stored in the message value field). The ability to dynamically provide different values without an a priori schema will be super useful.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] eric-maynard commented on pull request #39519: [SPARK-41995][SQL] Accept non-foldable expressions in schema_of_json
Posted by GitBox <gi...@apache.org>.
eric-maynard commented on PR #39519:
URL: https://github.com/apache/spark/pull/39519#issuecomment-1379685811
I see, where should we change to add to the other APIs? It looks like SchemaOfJson is used under the hood by the Scala API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #39519: [SPARK-41995][SQL] Accept non-foldable expressions in schema_of_json
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #39519:
URL: https://github.com/apache/spark/pull/39519#issuecomment-1379641861
Would be great if we can add it to Scala, Python, and R API. I don't mind doing that in a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #39519: [SPARK-41995][SQL] Accept non-foldable expressions in schema_of_json
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #39519:
URL: https://github.com/apache/spark/pull/39519#discussion_r1067581392
##########
sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala:
##########
@@ -583,6 +583,17 @@ class JsonFunctionsSuite extends QueryTest with SharedSparkSession {
checkAnswer(df, Seq(Row("STRUCT<a: BIGINT>")))
}
+ test("schema_of_json works on non-foldable columns") {
Review Comment:
```suggestion
test("SPARK-41995: schema_of_json works on non-foldable columns") {
```
##########
sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala:
##########
@@ -583,6 +583,17 @@ class JsonFunctionsSuite extends QueryTest with SharedSparkSession {
checkAnswer(df, Seq(Row("STRUCT<a: BIGINT>")))
}
+ test("schema_of_json works on non-foldable columns") {
+ val in = Seq("""{"a": [1, 2, 3]}""").toDS()
+ val out = in.select(schema_of_json($"value") as "parsed_schema")
+ checkAnswer(out, Row("STRUCT<a: ARRAY<BIGINT>>"))
+ }
+
+ test("schema_of_json returns null on null input") {
Review Comment:
```suggestion
test("SPARK-41995: schema_of_json returns null on null input") {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] eric-maynard commented on pull request #39519: [SPARK-41995][SQL] Accept non-foldable expressions in schema_of_json
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on PR #39519:
URL: https://github.com/apache/spark/pull/39519#issuecomment-1458985753
Closing for inactivity
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] eric-maynard closed pull request #39519: [SPARK-41995][SQL] Accept non-foldable expressions in schema_of_json
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard closed pull request #39519: [SPARK-41995][SQL] Accept non-foldable expressions in schema_of_json
URL: https://github.com/apache/spark/pull/39519
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #39519: [SPARK-41995][SQL] Accept non-foldable expressions in schema_of_json
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on PR #39519:
URL: https://github.com/apache/spark/pull/39519#issuecomment-1380641820
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #39519: [SPARK-41995] Accept non-foldable expressions in schema_of_json
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #39519:
URL: https://github.com/apache/spark/pull/39519#discussion_r1067580794
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala:
##########
@@ -793,27 +796,17 @@ case class SchemaOfJson(
@transient
private lazy val jsonInferSchema = new JsonInferSchema(jsonOptions)
- @transient
- private lazy val json = child.eval().asInstanceOf[UTF8String]
-
- override def checkInputDataTypes(): TypeCheckResult = {
- if (child.foldable && json != null) {
- super.checkInputDataTypes()
- } else if (!child.foldable) {
- DataTypeMismatch(
- errorSubClass = "NON_FOLDABLE_INPUT",
- messageParameters = Map(
- "inputName" -> "json",
- "inputType" -> toSQLType(child.dataType),
- "inputExpr" -> toSQLExpr(child)))
+ override def eval(v: InternalRow): Any = {
Review Comment:
Can we override `nullSafeEval` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org