You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by MaxGekk <gi...@git.apache.org> on 2018/07/01 10:03:09 UTC
[GitHub] spark pull request #21686: [SPARK-24709][SQL] schema_of_json() - schema infe...
GitHub user MaxGekk opened a pull request:
https://github.com/apache/spark/pull/21686
[SPARK-24709][SQL] schema_of_json() - schema inference from an example
## What changes were proposed in this pull request?
In the PR, I propose to add new function - *schema_of_json()* which infers schema of JSON string literal. The result of the function is a string containing a schema in DDL format.
One of the use cases is using of *schema_of_json()* in the combination with *from_json()*. Currently, _from_json()_ requires a schema as a mandatory argument. The *schema_of_json()* function will allow to point out an JSON string as an example which has the same schema as the first argument of _from_json()_. For instance:
```sql
select from_json(json_column, schema_of_json('{"c1": [0], "c2": [{"c2":0}]}'))
from json_table;
```
## How was this patch tested?
Added new test to `JsonFunctionsSuite`, `JsonExpressionsSuite` and SQL tests to `json-functions.sql`
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/MaxGekk/spark-1 infer_schema_json
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21686.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21686
----
commit 891f3ce6161a079f56a84ae3ff221996561d1506
Author: Maxim Gekk <ma...@...>
Date: 2018-06-30T18:12:00Z
Implemented new expression - SchemaOfJson
commit 26f3275bc65b8a97ad4be81bbfe1fc440fe73ff2
Author: Maxim Gekk <ma...@...>
Date: 2018-06-30T20:04:40Z
Fix imports
commit 1848a7ac4d0aa8fc95e964d0c3e9b893ae766fe3
Author: Maxim Gekk <ma...@...>
Date: 2018-06-30T20:05:49Z
from_json() accepts output of schema_of_json()
commit 42da3f24d63369a30944817f482818bca4917c88
Author: Maxim Gekk <ma...@...>
Date: 2018-06-30T20:06:09Z
Added sql tests
commit 97d93b3df7907c6501dd239e66b4ba3d5a4266c2
Author: Maxim Gekk <ma...@...>
Date: 2018-06-30T20:52:45Z
New functions - from_json which accepts a Column and schema_of_json
commit ab82bd81fdd4207d8b3321e37d31da424eba15f6
Author: Maxim Gekk <ma...@...>
Date: 2018-06-30T20:53:02Z
Tests for new functions
commit 174f8ab7d446b4f5df48d561fdc726708e5c4a52
Author: Maxim Gekk <ma...@...>
Date: 2018-06-30T21:00:00Z
Fix for json functions suite
commit d77ed456b6ee4b487212be9ba9a3497c0d72d374
Author: Maxim Gekk <ma...@...>
Date: 2018-06-30T22:03:41Z
Added the schema_of_json() function to PySpark
commit 086f6c17a5d9ec947d096990eaececbcd497c132
Author: Maxim Gekk <ma...@...>
Date: 2018-07-01T09:26:07Z
Fix schema_of_json() in PySpark
commit 2ff71e8d8d94c6d5777525a323611e95731f7dda
Author: Maxim Gekk <ma...@...>
Date: 2018-07-01T09:47:25Z
Adding ticket's number to test titles.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21686: [SPARK-24709][SQL] schema_of_json() - schema inference f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21686
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21686: [SPARK-24709][SQL] schema_of_json() - schema inference f...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/21686
Thanks. Awesome. This matches what I had in mind then.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21686: [SPARK-24709][SQL] schema_of_json() - schema infe...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199907218
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3381,6 +3381,48 @@ object functions {
from_json(e, dataType, options)
}
+ /**
+ * (Scala-specific) Parses a column containing a JSON string into a `MapType` with `StringType`
+ * as keys type, `StructType` or `ArrayType` of `StructType`s with the specified schema.
+ * Returns `null`, in the case of an unparseable string.
+ *
+ * @param e a string column containing JSON data.
+ * @param schema the schema to use when parsing the json string
+ *
+ * @group collection_funcs
+ * @since 2.4.0
+ */
+ def from_json(e: Column, schema: Column): Column = {
+ from_json(e, schema, Map.empty[String, String].asJava)
+ }
+
+ /**
+ * (Java-specific) Parses a column containing a JSON string into a `MapType` with `StringType`
+ * as keys type, `StructType` or `ArrayType` of `StructType`s with the specified schema.
+ * Returns `null`, in the case of an unparseable string.
+ *
+ * @param e a string column containing JSON data.
+ * @param schema the schema to use when parsing the json string
+ * @param options options to control how the json is parsed. accepts the same options and the
+ * json data source.
+ *
+ * @group collection_funcs
+ * @since 2.4.0
+ */
+ def from_json(e: Column, schema: Column, options: java.util.Map[String, String]): Column = {
--- End diff --
Ah, I see. I am fine then. Thanks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21686: [SPARK-24709][SQL] schema_of_json() - schema inference f...
Posted by MaxGekk <gi...@git.apache.org>.
Github user MaxGekk commented on the issue:
https://github.com/apache/spark/pull/21686
> Does this actually work in SQL?
Yes, it does. Please, have a look at the SQL test:
https://github.com/apache/spark/pull/21686/files#diff-3b8a538abd658a260aa32c4aa593bed7R41
> How does it work when we don't have a data type that's a schema?
We recently supported schema as any data type in DDL format: https://github.com/apache/spark/pull/21550 . The new function uses the same mechanism as we use for schema inferring when we read files. So, it can infer and return any data type (but not `MapType` for now) as string in DDL format. And `from_json()` can accept it due to the https://github.com/apache/spark/pull/21550 . @rxin If I didn't answer to your question, please, clarify what do you mean.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21686: [SPARK-24709][SQL] schema_of_json() - schema inference f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21686
**[Test build #92511 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92511/testReport)** for PR 21686 at commit [`56c925d`](https://github.com/apache/spark/commit/56c925d1585f6c509d1f6cb4b17bb6379e60c3fb).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21686: [SPARK-24709][SQL] schema_of_json() - schema inference f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21686
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92578/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21686: [SPARK-24709][SQL] schema_of_json() - schema infe...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199881408
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3381,6 +3381,48 @@ object functions {
from_json(e, dataType, options)
}
+ /**
+ * (Scala-specific) Parses a column containing a JSON string into a `MapType` with `StringType`
+ * as keys type, `StructType` or `ArrayType` of `StructType`s with the specified schema.
+ * Returns `null`, in the case of an unparseable string.
+ *
+ * @param e a string column containing JSON data.
+ * @param schema the schema to use when parsing the json string
+ *
+ * @group collection_funcs
+ * @since 2.4.0
+ */
+ def from_json(e: Column, schema: Column): Column = {
+ from_json(e, schema, Map.empty[String, String].asJava)
+ }
+
+ /**
+ * (Scala-specific) Parses a column containing a JSON string into a `MapType` with `StringType`
--- End diff --
Actually, can we remove this version for now and add it when it's requested? There is a concern about this file getting too long and `from_json` has so many variants.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21686: [SPARK-24709][SQL] schema_of_json() - schema infe...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199891460
--- Diff: python/pyspark/sql/functions.py ---
@@ -2189,11 +2189,16 @@ def from_json(col, schema, options={}):
>>> df = spark.createDataFrame(data, ("key", "value"))
>>> df.select(from_json(df.value, schema).alias("json")).collect()
[Row(json=[Row(a=1)])]
+ >>> schema = schema_of_json(lit('''{"a": 0}'''))
--- End diff --
feel free to fix other examples above too
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21686: [SPARK-24709][SQL] schema_of_json() - schema infe...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199890648
--- Diff: python/pyspark/sql/functions.py ---
@@ -2189,11 +2189,16 @@ def from_json(col, schema, options={}):
>>> df = spark.createDataFrame(data, ("key", "value"))
>>> df.select(from_json(df.value, schema).alias("json")).collect()
[Row(json=[Row(a=1)])]
+ >>> schema = schema_of_json(lit('''{"a": 0}'''))
--- End diff --
nit: `'''{"a": 0}'''` -> `'{"a": 0}'`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21686: [SPARK-24709][SQL] schema_of_json() - schema infe...
Posted by MaxGekk <gi...@git.apache.org>.
Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199883857
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
@@ -744,11 +747,42 @@ case class StructsToJson(
override def inputTypes: Seq[AbstractDataType] = TypeCollection(ArrayType, StructType) :: Nil
}
+/**
+ * A function infers schema of JSON string.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(json[, options]) - Returns schema in the DDL format of JSON string.",
+ examples = """
+ Examples:
+ > SELECT _FUNC_('[{"col":0}]');
+ array<struct<col:int>>
+ """)
+case class SchemaOfJson(child: Expression)
+ extends UnaryExpression with String2StringExpression with CodegenFallback {
+
+ private val jsonOptions = new JSONOptions(Map.empty, "UTC")
+ private val jsonFactory = new JsonFactory()
--- End diff --
Thank you for pointing this out. I really didn't know that I have to call the method.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21686: [SPARK-24709][SQL] schema_of_json() - schema infe...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199857808
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3381,6 +3381,48 @@ object functions {
from_json(e, dataType, options)
}
+ /**
+ * (Scala-specific) Parses a column containing a JSON string into a `MapType` with `StringType`
+ * as keys type, `StructType` or `ArrayType` of `StructType`s with the specified schema.
+ * Returns `null`, in the case of an unparseable string.
+ *
+ * @param e a string column containing JSON data.
+ * @param schema the schema to use when parsing the json string
+ *
+ * @group collection_funcs
+ * @since 2.4.0
+ */
+ def from_json(e: Column, schema: Column): Column = {
+ from_json(e, schema, Map.empty[String, String].asJava)
+ }
+
+ /**
+ * (Scala-specific) Parses a column containing a JSON string into a `MapType` with `StringType`
--- End diff --
Is it Scala specific or Java specific?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21686: [SPARK-24709][SQL] schema_of_json() - schema infe...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199856590
--- Diff: python/pyspark/sql/functions.py ---
@@ -2235,6 +2240,28 @@ def to_json(col, options={}):
return Column(jc)
+@ignore_unicode_prefix
+@since(2.4)
+def schema_of_json(col):
+ """
+ Parses a column containing a JSON string and infers its schema in DDL format.
+
+ :param col: string column in json format
+
+ >>> from pyspark.sql.types import *
+ >>> data = [(1, '''{"a": 1}''')]
+ >>> df = spark.createDataFrame(data, ("key", "value"))
+ >>> df.select(schema_of_json(df.value).alias("json")).collect()
+ [Row(json=u'struct<a:bigint>')]
+ >>> df.select(schema_of_json(lit('''{"a": 0}''')).alias("json")).collect()
--- End diff --
minor nit `'''{"a": 0}'''` -> `'{"a": 0}'`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21686: [SPARK-24709][SQL] schema_of_json() - schema inference f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21686
**[Test build #92575 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92575/testReport)** for PR 21686 at commit [`c993fd1`](https://github.com/apache/spark/commit/c993fd1df4a174f879943a1b4318c4dd9cfa68f7).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21686: [SPARK-24709][SQL] schema_of_json() - schema inference f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21686
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21686: [SPARK-24709][SQL] schema_of_json() - schema inference f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21686
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92511/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21686: [SPARK-24709][SQL] schema_of_json() - schema inference f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21686
**[Test build #92578 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92578/testReport)** for PR 21686 at commit [`86f6886`](https://github.com/apache/spark/commit/86f6886330ebcd7c3abf8b6d7b0725645f7db8aa).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21686: [SPARK-24709][SQL] schema_of_json() - schema infe...
Posted by MaxGekk <gi...@git.apache.org>.
Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199881316
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
@@ -744,11 +747,42 @@ case class StructsToJson(
override def inputTypes: Seq[AbstractDataType] = TypeCollection(ArrayType, StructType) :: Nil
}
+/**
+ * A function infers schema of JSON string.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(json[, options]) - Returns schema in the DDL format of JSON string.",
+ examples = """
+ Examples:
+ > SELECT _FUNC_('[{"col":0}]');
+ array<struct<col:int>>
+ """)
+case class SchemaOfJson(child: Expression)
+ extends UnaryExpression with String2StringExpression with CodegenFallback {
+
+ private val jsonOptions = new JSONOptions(Map.empty, "UTC")
+ private val jsonFactory = new JsonFactory()
+
+ override def convert(v: UTF8String): UTF8String = {
+ val dt = Utils.tryWithResource(CreateJacksonParser.utf8String(jsonFactory, v)) { parser =>
+ parser.nextToken()
+ inferField(parser, jsonOptions)
+ }
+
+ UTF8String.fromString(dt.catalogString)
+ }
+}
+
object JsonExprUtils {
- def validateSchemaLiteral(exp: Expression): DataType = exp match {
+ def evalSchemaExpr(exp: Expression): DataType = exp match {
case Literal(s, StringType) => DataType.fromDDL(s.toString)
- case e => throw new AnalysisException(s"Expected a string literal instead of $e")
+ case e @ SchemaOfJson(_: Literal) =>
+ val ddlSchema = e.eval().asInstanceOf[UTF8String]
+ DataType.fromDDL(ddlSchema.toString)
+ case e => throw new AnalysisException(
+ "Schema should be specified in DDL format as a string literal" +
+ s" or output of the schema_of_json function instead of $e")
--- End diff --
`exp` contains a schema or something like that. Maybe do you mean `$e` -> `${e.prettyName}`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21686: [SPARK-24709][SQL] schema_of_json() - schema inference f...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21686
Seems fine to me otherwise.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21686: [SPARK-24709][SQL] schema_of_json() - schema infe...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199842942
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
@@ -744,11 +747,42 @@ case class StructsToJson(
override def inputTypes: Seq[AbstractDataType] = TypeCollection(ArrayType, StructType) :: Nil
}
+/**
+ * A function infers schema of JSON string.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(json[, options]) - Returns schema in the DDL format of JSON string.",
+ examples = """
+ Examples:
+ > SELECT _FUNC_('[{"col":0}]');
+ array<struct<col:int>>
+ """)
--- End diff --
`since`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21686: [SPARK-24709][SQL] schema_of_json() - schema inference f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21686
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21686: [SPARK-24709][SQL] schema_of_json() - schema inference f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21686
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92575/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21686: [SPARK-24709][SQL] schema_of_json() - schema infe...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199891105
--- Diff: python/pyspark/sql/functions.py ---
@@ -2235,6 +2240,28 @@ def to_json(col, options={}):
return Column(jc)
+@ignore_unicode_prefix
+@since(2.4)
+def schema_of_json(col):
+ """
+ Parses a column containing a JSON string and infers its schema in DDL format.
+
+ :param col: string column in json format
+
+ >>> from pyspark.sql.types import *
+ >>> data = [(1, '''{"a": 1}''')]
--- End diff --
ditto
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21686: [SPARK-24709][SQL] schema_of_json() - schema inference f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21686
**[Test build #92575 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92575/testReport)** for PR 21686 at commit [`c993fd1`](https://github.com/apache/spark/commit/c993fd1df4a174f879943a1b4318c4dd9cfa68f7).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21686: [SPARK-24709][SQL] schema_of_json() - schema inference f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21686
**[Test build #92510 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92510/testReport)** for PR 21686 at commit [`2ff71e8`](https://github.com/apache/spark/commit/2ff71e8d8d94c6d5777525a323611e95731f7dda).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21686: [SPARK-24709][SQL] schema_of_json() - schema infe...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199857371
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3381,6 +3381,48 @@ object functions {
from_json(e, dataType, options)
}
+ /**
+ * (Scala-specific) Parses a column containing a JSON string into a `MapType` with `StringType`
+ * as keys type, `StructType` or `ArrayType` of `StructType`s with the specified schema.
+ * Returns `null`, in the case of an unparseable string.
+ *
+ * @param e a string column containing JSON data.
+ * @param schema the schema to use when parsing the json string
+ *
+ * @group collection_funcs
+ * @since 2.4.0
+ */
+ def from_json(e: Column, schema: Column): Column = {
+ from_json(e, schema, Map.empty[String, String].asJava)
+ }
+
+ /**
+ * (Scala-specific) Parses a column containing a JSON string into a `MapType` with `StringType`
+ * as keys type, `StructType` or `ArrayType` of `StructType`s with the specified schema.
+ * Returns `null`, in the case of an unparseable string.
+ *
+ * @param e a string column containing JSON data.
+ * @param schema the schema to use when parsing the json string
+ * @param options options to control how the json is parsed. accepts the same options and the
+ * json data source.
+ *
+ * @group collection_funcs
+ * @since 2.4.0
+ */
+ def from_json(e: Column, schema: Column, options: java.util.Map[String, String]): Column = {
+ withExpr {new JsonToStructs(e.expr, schema.expr, options.asScala.toMap)}
--- End diff --
`{n` -> `{ n`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21686: [SPARK-24709][SQL] schema_of_json() - schema inference f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21686
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21686: [SPARK-24709][SQL] schema_of_json() - schema inference f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21686
**[Test build #92580 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92580/testReport)** for PR 21686 at commit [`dc35731`](https://github.com/apache/spark/commit/dc35731f4cd99c3687be486fcf15e9f9883cb139).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21686: [SPARK-24709][SQL] schema_of_json() - schema infe...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199826429
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
@@ -744,11 +747,42 @@ case class StructsToJson(
override def inputTypes: Seq[AbstractDataType] = TypeCollection(ArrayType, StructType) :: Nil
}
+/**
+ * A function infers schema of JSON string.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(json[, options]) - Returns schema in the DDL format of JSON string.",
+ examples = """
+ Examples:
+ > SELECT _FUNC_('[{"col":0}]');
+ array<struct<col:int>>
+ """)
+case class SchemaOfJson(child: Expression)
+ extends UnaryExpression with String2StringExpression with CodegenFallback {
+
+ private val jsonOptions = new JSONOptions(Map.empty, "UTC")
+ private val jsonFactory = new JsonFactory()
+
+ override def convert(v: UTF8String): UTF8String = {
+ val dt = Utils.tryWithResource(CreateJacksonParser.utf8String(jsonFactory, v)) { parser =>
+ parser.nextToken()
+ inferField(parser, jsonOptions)
+ }
+
+ UTF8String.fromString(dt.catalogString)
+ }
+}
+
object JsonExprUtils {
- def validateSchemaLiteral(exp: Expression): DataType = exp match {
+ def evalSchemaExpr(exp: Expression): DataType = exp match {
case Literal(s, StringType) => DataType.fromDDL(s.toString)
- case e => throw new AnalysisException(s"Expected a string literal instead of $e")
+ case e @ SchemaOfJson(_: Literal) =>
+ val ddlSchema = e.eval().asInstanceOf[UTF8String]
+ DataType.fromDDL(ddlSchema.toString)
+ case e => throw new AnalysisException(
+ "Schema should be specified in DDL format as a string literal" +
+ s" or output of the schema_of_json function instead of $e")
--- End diff --
minor nit: `schema_of_json ` -> `exp.prettyName`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21686: [SPARK-24709][SQL] schema_of_json() - schema infe...
Posted by MaxGekk <gi...@git.apache.org>.
Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199899708
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3381,6 +3381,48 @@ object functions {
from_json(e, dataType, options)
}
+ /**
+ * (Scala-specific) Parses a column containing a JSON string into a `MapType` with `StringType`
+ * as keys type, `StructType` or `ArrayType` of `StructType`s with the specified schema.
+ * Returns `null`, in the case of an unparseable string.
+ *
+ * @param e a string column containing JSON data.
+ * @param schema the schema to use when parsing the json string
+ *
+ * @group collection_funcs
+ * @since 2.4.0
+ */
+ def from_json(e: Column, schema: Column): Column = {
+ from_json(e, schema, Map.empty[String, String].asJava)
+ }
+
+ /**
+ * (Java-specific) Parses a column containing a JSON string into a `MapType` with `StringType`
+ * as keys type, `StructType` or `ArrayType` of `StructType`s with the specified schema.
+ * Returns `null`, in the case of an unparseable string.
+ *
+ * @param e a string column containing JSON data.
+ * @param schema the schema to use when parsing the json string
+ * @param options options to control how the json is parsed. accepts the same options and the
+ * json data source.
+ *
+ * @group collection_funcs
+ * @since 2.4.0
+ */
+ def from_json(e: Column, schema: Column, options: java.util.Map[String, String]): Column = {
--- End diff --
we call the method from python: https://github.com/apache/spark/pull/21686/files#diff-f5295f69bfbdbf6e161aed54057ea36dR2202
Do you really want to revert changes for python?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21686: [SPARK-24709][SQL] schema_of_json() - schema inference f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21686
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21686: [SPARK-24709][SQL] schema_of_json() - schema infe...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199890912
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3381,6 +3381,48 @@ object functions {
from_json(e, dataType, options)
}
+ /**
+ * (Scala-specific) Parses a column containing a JSON string into a `MapType` with `StringType`
+ * as keys type, `StructType` or `ArrayType` of `StructType`s with the specified schema.
+ * Returns `null`, in the case of an unparseable string.
+ *
+ * @param e a string column containing JSON data.
+ * @param schema the schema to use when parsing the json string
+ *
+ * @group collection_funcs
+ * @since 2.4.0
+ */
+ def from_json(e: Column, schema: Column): Column = {
+ from_json(e, schema, Map.empty[String, String].asJava)
+ }
+
+ /**
+ * (Java-specific) Parses a column containing a JSON string into a `MapType` with `StringType`
+ * as keys type, `StructType` or `ArrayType` of `StructType`s with the specified schema.
+ * Returns `null`, in the case of an unparseable string.
+ *
+ * @param e a string column containing JSON data.
+ * @param schema the schema to use when parsing the json string
+ * @param options options to control how the json is parsed. accepts the same options and the
+ * json data source.
+ *
+ * @group collection_funcs
+ * @since 2.4.0
+ */
+ def from_json(e: Column, schema: Column, options: java.util.Map[String, String]): Column = {
--- End diff --
Let me leave my last comment, https://github.com/apache/spark/pull/21686#discussion_r199881408 in case it's missed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21686: [SPARK-24709][SQL] schema_of_json() - schema infe...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199881050
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3381,6 +3381,48 @@ object functions {
from_json(e, dataType, options)
}
+ /**
+ * (Scala-specific) Parses a column containing a JSON string into a `MapType` with `StringType`
+ * as keys type, `StructType` or `ArrayType` of `StructType`s with the specified schema.
+ * Returns `null`, in the case of an unparseable string.
+ *
+ * @param e a string column containing JSON data.
+ * @param schema the schema to use when parsing the json string
+ *
+ * @group collection_funcs
+ * @since 2.4.0
+ */
+ def from_json(e: Column, schema: Column): Column = {
+ from_json(e, schema, Map.empty[String, String].asJava)
+ }
+
+ /**
+ * (Scala-specific) Parses a column containing a JSON string into a `MapType` with `StringType`
--- End diff --
Yup. That's because Java users will more likely use Java's collections rather than Scala's collection which works weirdly in Java side.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21686: [SPARK-24709][SQL] schema_of_json() - schema inference f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21686
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92510/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21686: [SPARK-24709][SQL] schema_of_json() - schema infe...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199879761
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3381,6 +3381,48 @@ object functions {
from_json(e, dataType, options)
}
+ /**
+ * (Scala-specific) Parses a column containing a JSON string into a `MapType` with `StringType`
+ * as keys type, `StructType` or `ArrayType` of `StructType`s with the specified schema.
+ * Returns `null`, in the case of an unparseable string.
+ *
+ * @param e a string column containing JSON data.
+ * @param schema the schema to use when parsing the json string
+ *
+ * @group collection_funcs
+ * @since 2.4.0
+ */
+ def from_json(e: Column, schema: Column): Column = {
+ from_json(e, schema, Map.empty[String, String].asJava)
+ }
+
+ /**
+ * (Scala-specific) Parses a column containing a JSON string into a `MapType` with `StringType`
+ * as keys type, `StructType` or `ArrayType` of `StructType`s with the specified schema.
+ * Returns `null`, in the case of an unparseable string.
+ *
+ * @param e a string column containing JSON data.
+ * @param schema the schema to use when parsing the json string
+ * @param options options to control how the json is parsed. accepts the same options and the
+ * json data source.
+ *
+ * @group collection_funcs
+ * @since 2.4.0
+ */
+ def from_json(e: Column, schema: Column, options: java.util.Map[String, String]): Column = {
+ withExpr {new JsonToStructs(e.expr, schema.expr, options.asScala.toMap)}
--- End diff --
Yup
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21686: [SPARK-24709][SQL] schema_of_json() - schema inference f...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/21686
Does this actually work in SQL? How does it work when we don't have a data type that's a schema?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21686: [SPARK-24709][SQL] schema_of_json() - schema infe...
Posted by MaxGekk <gi...@git.apache.org>.
Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199878616
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3381,6 +3381,48 @@ object functions {
from_json(e, dataType, options)
}
+ /**
+ * (Scala-specific) Parses a column containing a JSON string into a `MapType` with `StringType`
+ * as keys type, `StructType` or `ArrayType` of `StructType`s with the specified schema.
+ * Returns `null`, in the case of an unparseable string.
+ *
+ * @param e a string column containing JSON data.
+ * @param schema the schema to use when parsing the json string
+ *
+ * @group collection_funcs
+ * @since 2.4.0
+ */
+ def from_json(e: Column, schema: Column): Column = {
+ from_json(e, schema, Map.empty[String, String].asJava)
+ }
+
+ /**
+ * (Scala-specific) Parses a column containing a JSON string into a `MapType` with `StringType`
--- End diff --
I am not sure about this note at all. Why should it be Java or Scala specific?
I will change it to `Java-specific` to have it in the same style as other comments.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21686: [SPARK-24709][SQL] schema_of_json() - schema inference f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21686
**[Test build #92580 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92580/testReport)** for PR 21686 at commit [`dc35731`](https://github.com/apache/spark/commit/dc35731f4cd99c3687be486fcf15e9f9883cb139).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21686: [SPARK-24709][SQL] schema_of_json() - schema infe...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199825750
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
@@ -744,11 +747,42 @@ case class StructsToJson(
override def inputTypes: Seq[AbstractDataType] = TypeCollection(ArrayType, StructType) :: Nil
}
+/**
+ * A function infers schema of JSON string.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(json[, options]) - Returns schema in the DDL format of JSON string.",
+ examples = """
+ Examples:
+ > SELECT _FUNC_('[{"col":0}]');
+ array<struct<col:int>>
+ """)
+case class SchemaOfJson(child: Expression)
+ extends UnaryExpression with String2StringExpression with CodegenFallback {
+
+ private val jsonOptions = new JSONOptions(Map.empty, "UTC")
+ private val jsonFactory = new JsonFactory()
--- End diff --
Seems `jsonOptions.setJacksonOptions(factory)` is missed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21686: [SPARK-24709][SQL] schema_of_json() - schema infe...
Posted by MaxGekk <gi...@git.apache.org>.
Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199877744
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3381,6 +3381,48 @@ object functions {
from_json(e, dataType, options)
}
+ /**
+ * (Scala-specific) Parses a column containing a JSON string into a `MapType` with `StringType`
+ * as keys type, `StructType` or `ArrayType` of `StructType`s with the specified schema.
+ * Returns `null`, in the case of an unparseable string.
+ *
+ * @param e a string column containing JSON data.
+ * @param schema the schema to use when parsing the json string
+ *
+ * @group collection_funcs
+ * @since 2.4.0
+ */
+ def from_json(e: Column, schema: Column): Column = {
+ from_json(e, schema, Map.empty[String, String].asJava)
+ }
+
+ /**
+ * (Scala-specific) Parses a column containing a JSON string into a `MapType` with `StringType`
+ * as keys type, `StructType` or `ArrayType` of `StructType`s with the specified schema.
+ * Returns `null`, in the case of an unparseable string.
+ *
+ * @param e a string column containing JSON data.
+ * @param schema the schema to use when parsing the json string
+ * @param options options to control how the json is parsed. accepts the same options and the
+ * json data source.
+ *
+ * @group collection_funcs
+ * @since 2.4.0
+ */
+ def from_json(e: Column, schema: Column, options: java.util.Map[String, String]): Column = {
+ withExpr {new JsonToStructs(e.expr, schema.expr, options.asScala.toMap)}
--- End diff --
Maybe just `withExpr(new JsonToStructs(...))`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21686: [SPARK-24709][SQL] schema_of_json() - schema infe...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199902249
--- Diff: python/pyspark/sql/functions.py ---
@@ -2189,11 +2189,16 @@ def from_json(col, schema, options={}):
>>> df = spark.createDataFrame(data, ("key", "value"))
>>> df.select(from_json(df.value, schema).alias("json")).collect()
[Row(json=[Row(a=1)])]
+ >>> schema = schema_of_json(lit('''{"a": 0}'''))
--- End diff --
Nope, I mean the examples here in this function.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21686: [SPARK-24709][SQL] schema_of_json() - schema inference f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21686
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92580/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21686: [SPARK-24709][SQL] schema_of_json() - schema inference f...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21686
@rxin, does this look okay to you? If so will check closely and get this in.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21686: [SPARK-24709][SQL] schema_of_json() - schema infe...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/21686
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21686: [SPARK-24709][SQL] schema_of_json() - schema inference f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21686
**[Test build #92578 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92578/testReport)** for PR 21686 at commit [`86f6886`](https://github.com/apache/spark/commit/86f6886330ebcd7c3abf8b6d7b0725645f7db8aa).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21686: [SPARK-24709][SQL] schema_of_json() - schema inference f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21686
**[Test build #92510 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92510/testReport)** for PR 21686 at commit [`2ff71e8`](https://github.com/apache/spark/commit/2ff71e8d8d94c6d5777525a323611e95731f7dda).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21686: [SPARK-24709][SQL] schema_of_json() - schema infe...
Posted by MaxGekk <gi...@git.apache.org>.
Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199894916
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3381,6 +3381,48 @@ object functions {
from_json(e, dataType, options)
}
+ /**
+ * (Scala-specific) Parses a column containing a JSON string into a `MapType` with `StringType`
+ * as keys type, `StructType` or `ArrayType` of `StructType`s with the specified schema.
+ * Returns `null`, in the case of an unparseable string.
+ *
+ * @param e a string column containing JSON data.
+ * @param schema the schema to use when parsing the json string
+ *
+ * @group collection_funcs
+ * @since 2.4.0
+ */
+ def from_json(e: Column, schema: Column): Column = {
+ from_json(e, schema, Map.empty[String, String].asJava)
+ }
+
+ /**
+ * (Scala-specific) Parses a column containing a JSON string into a `MapType` with `StringType`
--- End diff --
Maybe we will convert `object functions` to `package object functions` and put json related methods to a separate file. This `functions object` becomes really big.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21686: [SPARK-24709][SQL] schema_of_json() - schema inference f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21686
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21686: [SPARK-24709][SQL] schema_of_json() - schema inference f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21686
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21686: [SPARK-24709][SQL] schema_of_json() - schema inference f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21686
**[Test build #92511 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92511/testReport)** for PR 21686 at commit [`56c925d`](https://github.com/apache/spark/commit/56c925d1585f6c509d1f6cb4b17bb6379e60c3fb).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21686: [SPARK-24709][SQL] schema_of_json() - schema inference f...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21686
Merged to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21686: [SPARK-24709][SQL] schema_of_json() - schema infe...
Posted by MaxGekk <gi...@git.apache.org>.
Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21686#discussion_r199899927
--- Diff: python/pyspark/sql/functions.py ---
@@ -2189,11 +2189,16 @@ def from_json(col, schema, options={}):
>>> df = spark.createDataFrame(data, ("key", "value"))
>>> df.select(from_json(df.value, schema).alias("json")).collect()
[Row(json=[Row(a=1)])]
+ >>> schema = schema_of_json(lit('''{"a": 0}'''))
--- End diff --
Do you mean for other functions too?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org