You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by goldmedal <gi...@git.apache.org> on 2017/08/07 18:53:56 UTC
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
GitHub user goldmedal opened a pull request:
https://github.com/apache/spark/pull/18875
[SPARK-21513][SQL] Allow UDF to_json support converting MapType to json
# What changes were proposed in this pull request?
UDF to_json only supports converting `StructType` or `ArrayType` of `StructType`s to a json output string now.
According to the discussion of JIRA SPARK-21513, I allow to `to_json` support converting `MapType<DataType, StructType>` to a json output string.
# How was this patch tested?
Adding unit test case.
cc @viirya @HyukjinKwon
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/goldmedal/spark SPARK-21513
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18875.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18875
----
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80984/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81454/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r138077808
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala ---
@@ -180,10 +180,30 @@ class JsonFunctionsSuite extends QueryTest with SharedSQLContext {
test("to_json - array") {
val df = Seq(Tuple1(Tuple1(1) :: Nil)).toDF("a")
+ val df2 = Seq(Tuple1(Map("a" -> 1) :: Nil)).toDF("a")
checkAnswer(
df.select(to_json($"a")),
Row("""[{"_1":1}]""") :: Nil)
+ checkAnswer(
+ df2.select(to_json($"a")),
+ Row("""[{"a":1}]""") :: Nil)
+ }
+
+ test("to_json - map") {
+ val df1 = Seq(Map("a" -> Tuple1(1))).toDF("a")
+ val df2 = Seq(Map(Tuple1(1) -> Tuple1(1))).toDF("a")
+ val df3 = Seq(Map("a" -> 1)).toDF("a")
+
+ checkAnswer(
+ df1.select(to_json($"a")),
+ Row("""{"a":{"_1":1}}""") :: Nil)
+ checkAnswer(
+ df2.select(to_json($"a")),
+ Row("""{"[0,1]":{"_1":1}}""") :: Nil)
--- End diff --
This case looks rather a string representation from `UnsafeRow` and a bug. Let's remove this test case for now and fix it later together in another PR.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81555 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81555/testReport)** for PR 18875 at commit [`723cd20`](https://github.com/apache/spark/commit/723cd20fbd4ad4ef6be513546f17f4629dcf3238).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81630 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81630/testReport)** for PR 18875 at commit [`b71b56a`](https://github.com/apache/spark/commit/b71b56ab49ba048f397459a18f946422d9440b29).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80532/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on the issue:
https://github.com/apache/spark/pull/18875
@HyukjinKwon OK, I'll work on R and Python. My JIRA id is 'goldmedal', too. Thanks for your review :)
@viirya Thanks for your mentor and review :)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r138095618
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala ---
@@ -180,10 +180,30 @@ class JsonFunctionsSuite extends QueryTest with SharedSQLContext {
test("to_json - array") {
val df = Seq(Tuple1(Tuple1(1) :: Nil)).toDF("a")
+ val df2 = Seq(Tuple1(Map("a" -> 1) :: Nil)).toDF("a")
checkAnswer(
df.select(to_json($"a")),
Row("""[{"_1":1}]""") :: Nil)
+ checkAnswer(
+ df2.select(to_json($"a")),
+ Row("""[{"a":1}]""") :: Nil)
+ }
+
+ test("to_json - map") {
+ val df1 = Seq(Map("a" -> Tuple1(1))).toDF("a")
+ val df2 = Seq(Map(Tuple1(1) -> Tuple1(1))).toDF("a")
+ val df3 = Seq(Map("a" -> 1)).toDF("a")
+
+ checkAnswer(
+ df1.select(to_json($"a")),
+ Row("""{"a":{"_1":1}}""") :: Nil)
+ checkAnswer(
+ df2.select(to_json($"a")),
+ Row("""{"[0,1]":{"_1":1}}""") :: Nil)
--- End diff --
I think we don't have a `toString` method for `UnsafeRow` to show up well-format string representation. The current `toString` for it is just for debugging. It simply gets the hex string for each 8 bytes. Consider the nature of `UnsafeRow`, this `toString` might make sense to me.
We can remove this test case if the result could be confusing, otherwise we should add a comment to explain why the JSON looks like it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81414/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81620 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81620/testReport)** for PR 18875 at commit [`b8ad442`](https://github.com/apache/spark/commit/b8ad442437390611c63d8b0705d02c3ca3ba3c9c).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on the issue:
https://github.com/apache/spark/pull/18875
@HyukjinKwon @viirya
Sorry for updating this PR so late. Please take a look when you are available. Thanks :)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81588 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81588/testReport)** for PR 18875 at commit [`b9a8c47`](https://github.com/apache/spark/commit/b9a8c4759fb6189005c02f30bb4108c28b941335).
* This patch passes all tests.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r132613088
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
@@ -659,13 +660,19 @@ case class StructsToJson(
(arr: Any) =>
gen.write(arr.asInstanceOf[ArrayData])
getAndReset()
+ case MapType(_: DataType, _: StructType, _: Boolean) =>
+ (map: Any) =>
+ val mapType = child.dataType.asInstanceOf[MapType]
+ gen.write(map.asInstanceOf[MapData], mapType)
+ getAndReset()
}
}
override def dataType: DataType = StringType
override def checkInputDataTypes(): TypeCheckResult = child.dataType match {
- case _: StructType | ArrayType(_: StructType, _) =>
+ case _: StructType | ArrayType(_: StructType, _) |
+ MapType(_: DataType, _: StructType, _: Boolean) =>
--- End diff --
I am quite sure we keep compatibility in the name although it might be best to not change if possible. I think we can also add `prettyName` to `to_json` for the column name concern too.
What do you think about renaming this in another minor PR or a followup just for less line diff, less complexity and for an easier review, considering this is his very first PR? I am also fine with doing this here if the opposite makes more sense to you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r138058695
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -26,20 +26,53 @@ import org.apache.spark.sql.catalyst.expressions.SpecializedGetters
import org.apache.spark.sql.catalyst.util.{ArrayData, DateTimeUtils, MapData}
import org.apache.spark.sql.types._
+/**
+ * `JackGenerator` can only be initialized with a `StructType` or a `MapType`.
+ * Once it is initialized with `StructType`, it can be used to write out a struct or an array of
+ * struct. Once it is initialized with ``MapType``, it can be used to write out a map or an array
+ * of map. An exception will be thrown if trying to write out a struct if it is initialized with
+ * a `MapType`, and vice verse.
+ */
private[sql] class JacksonGenerator(
- schema: StructType,
+ dataType: DataType,
writer: Writer,
options: JSONOptions) {
// A `ValueWriter` is responsible for writing a field of an `InternalRow` to appropriate
// JSON data. Here we are using `SpecializedGetters` rather than `InternalRow` so that
// we can directly access data in `ArrayData` without the help of `SpecificMutableRow`.
private type ValueWriter = (SpecializedGetters, Int) => Unit
+ // `JackGenerator` can only be initialized with a `StructType` or a `MapType`.
+ dataType match {
--- End diff --
I'd just do a `require(... .isInstanceOf... || ..., "...")` instead.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137986124
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
@@ -677,14 +696,42 @@ case class StructsToJson(
override def checkInputDataTypes(): TypeCheckResult = child.dataType match {
case _: StructType | ArrayType(_: StructType, _) =>
try {
- JacksonUtils.verifySchema(rowSchema)
+ JacksonUtils.verifySchema(rowSchema.asInstanceOf[StructType])
+ TypeCheckResult.TypeCheckSuccess
+ } catch {
+ case e: UnsupportedOperationException =>
+ TypeCheckResult.TypeCheckFailure(e.getMessage)
+ }
+ case ArrayType(mt: MapType, _) =>
+ try {
+ val st = StructType(StructField("a", mt) :: Nil)
+ JacksonUtils.verifySchema(st)
+ TypeCheckResult.TypeCheckSuccess
+ } catch {
+ case e: UnsupportedOperationException =>
+ TypeCheckResult.TypeCheckFailure(e.getMessage)
+ }
+ case MapType(_: DataType, st: StructType, _: Boolean) =>
--- End diff --
Yeah, as you support arbitrary map. Not all map types actually, it leaves to `JacksonUtils` to verify the schema.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81458 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81458/testReport)** for PR 18875 at commit [`9972ed9`](https://github.com/apache/spark/commit/9972ed98fd170924cacc2efa9397b214078892f5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #80585 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80585/testReport)** for PR 18875 at commit [`3045771`](https://github.com/apache/spark/commit/30457716075a4b061df9909b26bc427fb35cf29e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #80585 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80585/testReport)** for PR 18875 at commit [`3045771`](https://github.com/apache/spark/commit/30457716075a4b061df9909b26bc427fb35cf29e).
* This patch **fails Spark unit tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r132614716
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
@@ -659,13 +660,19 @@ case class StructsToJson(
(arr: Any) =>
gen.write(arr.asInstanceOf[ArrayData])
getAndReset()
+ case MapType(_: DataType, _: StructType, _: Boolean) =>
+ (map: Any) =>
+ val mapType = child.dataType.asInstanceOf[MapType]
+ gen.write(map.asInstanceOf[MapData], mapType)
+ getAndReset()
}
}
override def dataType: DataType = StringType
override def checkInputDataTypes(): TypeCheckResult = child.dataType match {
- case _: StructType | ArrayType(_: StructType, _) =>
+ case _: StructType | ArrayType(_: StructType, _) |
--- End diff --
If we support `MapType` in general, I think `ArrayType(MapType)` will be allowed? We don't need to address it in this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81555/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137285989
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
@@ -669,14 +679,25 @@ case class StructsToJson(
override def checkInputDataTypes(): TypeCheckResult = child.dataType match {
case _: StructType | ArrayType(_: StructType, _) =>
try {
- JacksonUtils.verifySchema(rowSchema)
+ JacksonUtils.verifySchema(rowSchema.asInstanceOf[StructType])
+ TypeCheckResult.TypeCheckSuccess
+ } catch {
+ case e: UnsupportedOperationException =>
+ TypeCheckResult.TypeCheckFailure(e.getMessage)
+ }
+ case MapType(_: DataType, st: StructType, _: Boolean) =>
+ try {
+ JacksonUtils.verifySchema(st)
TypeCheckResult.TypeCheckSuccess
} catch {
case e: UnsupportedOperationException =>
TypeCheckResult.TypeCheckFailure(e.getMessage)
}
+ case _: MapType =>
+ TypeCheckResult.TypeCheckSuccess
--- End diff --
ok. thanks for review =)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r138233095
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/json/JacksonGeneratorSuite.scala ---
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.json
+
+import java.io.CharArrayWriter
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, DateTimeUtils, GenericArrayData}
+import org.apache.spark.sql.types._
+
+class JacksonGeneratorSuite extends SparkFunSuite {
+
+ val gmtId = DateTimeUtils.TimeZoneGMT.getID
+ val option = new JSONOptions(Map.empty, gmtId)
+
+ test("initial with StructType and write out a row") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = InternalRow(1)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """{"a":1}""")
+ }
+
+ test("initial with StructType and write out rows") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = new GenericArrayData(InternalRow(1) :: InternalRow(2) :: Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """[{"a":1},{"a":2}]""")
+ }
+
+ test("initial with StructType and write out an array with single empty row") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = new GenericArrayData(InternalRow(null) :: Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """[{}]""")
+ }
+
+ test("initial with StructType and write out an empty array") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = new GenericArrayData(Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """[]""")
+ }
+
+ test("initial with Map and write out a map data") {
+ val dataType = MapType(StringType, IntegerType)
+ val input = ArrayBasedMapData(Map("a" -> 1))
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """{"a":1}""")
+ }
+
+ test("initial with Map and write out an array of maps") {
+ val dataType = MapType(StringType, IntegerType)
+ val input = new GenericArrayData(
+ ArrayBasedMapData(Map("a" -> 1)) :: ArrayBasedMapData(Map("b" -> 2)) :: Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """[{"a":1},{"b":2}]""")
+ }
+
+ test("error handling: initial with StructType but error calling write a map") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = ArrayBasedMapData(Map("a" -> 1))
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ intercept[Exception] {
+ gen.write(input)
+ }
+ }
+
+ test("error handling: initial with StructType but error calling write an array of maps") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = new GenericArrayData(
+ ArrayBasedMapData(Map("a" -> 1)) :: ArrayBasedMapData(Map("b" -> 2)) :: Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ intercept[Exception] {
--- End diff --
Actually since this is for internal use, and I assume the callers know what they are doing and provide correct arrays. But we can add a TODO comment for it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r138237482
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -193,14 +223,35 @@ private[sql] class JacksonGenerator(
*
* @param row The row to convert
*/
- def write(row: InternalRow): Unit = writeObject(writeFields(row, schema, rootFieldWriters))
+ def write(row: InternalRow): Unit = {
+ writeObject(writeFields(
+ fieldWriters = rootFieldWriters,
+ row = row,
+ schema = dataType.asInstanceOf[StructType]))
+ }
+
+
+ /**
+ * Transforms multiple `InternalRow`s or `MapData`s to JSON array using Jackson
+ *
+ * @param array The array of rows or maps to convert
+ */
+ def write(array: ArrayData): Unit = writeArray(writeArrayData(
+ fieldWriter = arrElementWriter,
+ array = array
+ ))
/**
- * Transforms multiple `InternalRow`s to JSON array using Jackson
+ * Transforms a single `MapData` to JSON object using Jackson
*
- * @param array The array of rows to convert
+ * @param map a map to convert
*/
- def write(array: ArrayData): Unit = writeArray(writeArrayData(array, arrElementWriter))
+ def write(map: MapData): Unit = {
+ writeObject(writeMapData(
+ fieldWriter = mapElementWriter,
--- End diff --
Ok Thanks for review :) I'll update it tonight.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/18875
Minor comments left, otherwise LGTM.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18875
Merged to master.
This was a big work for the very first contribution. Thanks for working on this and bearing with me @goldmedal and @viirya.
Would you like to work on R and Python too? I think the works should be quite small.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81537 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81537/testReport)** for PR 18875 at commit [`36ce961`](https://github.com/apache/spark/commit/36ce9614c078c9c0aca62a672948d8581b43e2ea).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18875
Thanks @goldmedal for this work and @viirya for careful review. Will take a close look too soon.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #80399 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80399/testReport)** for PR 18875 at commit [`4cf4212`](https://github.com/apache/spark/commit/4cf42127a7c1044263a69bc7c2562f198558eb27).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137979014
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
@@ -677,14 +696,27 @@ case class StructsToJson(
override def checkInputDataTypes(): TypeCheckResult = child.dataType match {
case _: StructType | ArrayType(_: StructType, _) =>
try {
- JacksonUtils.verifySchema(rowSchema)
+ JacksonUtils.verifySchema(rowSchema.asInstanceOf[StructType])
+ TypeCheckResult.TypeCheckSuccess
+ } catch {
+ case e: UnsupportedOperationException =>
+ TypeCheckResult.TypeCheckFailure(e.getMessage)
+ }
+ case ArrayType(_: MapType, _) =>
+ TypeCheckResult.TypeCheckSuccess
+ case MapType(_: DataType, st: StructType, _: Boolean) =>
+ try {
+ JacksonUtils.verifySchema(st)
TypeCheckResult.TypeCheckSuccess
} catch {
case e: UnsupportedOperationException =>
TypeCheckResult.TypeCheckFailure(e.getMessage)
}
+ case _: MapType =>
--- End diff --
oh yes. It's a tricky way but make sense. =D
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/18875
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81461/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81458 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81458/testReport)** for PR 18875 at commit [`9972ed9`](https://github.com/apache/spark/commit/9972ed98fd170924cacc2efa9397b214078892f5).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r131932270
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
@@ -659,13 +660,19 @@ case class StructsToJson(
(arr: Any) =>
gen.write(arr.asInstanceOf[ArrayData])
getAndReset()
+ case MapType(_: DataType, _: StructType, _: Boolean) =>
+ (map: Any) =>
+ val mapType = child.dataType.asInstanceOf[MapType]
+ gen.write(map.asInstanceOf[MapData], mapType)
+ getAndReset()
}
}
override def dataType: DataType = StringType
override def checkInputDataTypes(): TypeCheckResult = child.dataType match {
- case _: StructType | ArrayType(_: StructType, _) =>
+ case _: StructType | ArrayType(_: StructType, _) |
+ MapType(_: DataType, _: StructType, _: Boolean) =>
--- End diff --
Oh, no. Similar to the ArrayType case, as this expression is named `StructsToJson`, I guess it is restricted to StructType?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81632 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81632/testReport)** for PR 18875 at commit [`8835aca`](https://github.com/apache/spark/commit/8835aca5a5ccf420010b24e43189bfa8f1e8dc2a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81643/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81624 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81624/testReport)** for PR 18875 at commit [`f567679`](https://github.com/apache/spark/commit/f567679ec724052266587e35cdf25868587de750).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81546/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137975214
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -193,14 +228,30 @@ private[sql] class JacksonGenerator(
*
* @param row The row to convert
*/
- def write(row: InternalRow): Unit = writeObject(writeFields(row, schema, rootFieldWriters))
+ def write(row: InternalRow): Unit = dataType match {
+ case st: StructType =>
+ writeObject(writeFields(row, st, rootFieldWriters))
+ case _ => throw new UnsupportedOperationException(
+ s"`JacksonGenerator` can only be used to write out a row when initialized with `StructType`.")
+ }
/**
- * Transforms multiple `InternalRow`s to JSON array using Jackson
+ * Transforms multiple `InternalRow`s or `MapData`s to JSON array using Jackson
*
- * @param array The array of rows to convert
+ * @param array The array of rows or maps to convert
*/
def write(array: ArrayData): Unit = writeArray(writeArrayData(array, arrElementWriter))
+ /**
+ * Transforms a `MapData` to JSON object using Jackson
+ *
+ * @param map a map to convert
+ */
+ def write(map: MapData): Unit = dataType match {
+ case mt: MapType => writeObject(writeMapData(map, mt, mapElementWriter))
+ case _ => throw new UnsupportedOperationException(
--- End diff --
Similar to https://github.com/apache/spark/pull/18875/files#r137975134, we can avoid this. Doing pattern matching per writing seems too burden.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81643 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81643/testReport)** for PR 18875 at commit [`0f2dd18`](https://github.com/apache/spark/commit/0f2dd186b48b2b46ed1367ebae811b5767ef8598).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #80372 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80372/testReport)** for PR 18875 at commit [`c3d4a09`](https://github.com/apache/spark/commit/c3d4a095b82091e986a46251ad5dc90b41166e68).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137979202
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
@@ -677,14 +696,27 @@ case class StructsToJson(
override def checkInputDataTypes(): TypeCheckResult = child.dataType match {
case _: StructType | ArrayType(_: StructType, _) =>
try {
- JacksonUtils.verifySchema(rowSchema)
+ JacksonUtils.verifySchema(rowSchema.asInstanceOf[StructType])
+ TypeCheckResult.TypeCheckSuccess
+ } catch {
+ case e: UnsupportedOperationException =>
+ TypeCheckResult.TypeCheckFailure(e.getMessage)
+ }
+ case ArrayType(_: MapType, _) =>
+ TypeCheckResult.TypeCheckSuccess
+ case MapType(_: DataType, st: StructType, _: Boolean) =>
+ try {
+ JacksonUtils.verifySchema(st)
TypeCheckResult.TypeCheckSuccess
} catch {
case e: UnsupportedOperationException =>
TypeCheckResult.TypeCheckFailure(e.getMessage)
}
+ case _: MapType =>
--- End diff --
Basically I don't like to increase the size of this PR anymore.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r131952135
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
@@ -659,13 +660,19 @@ case class StructsToJson(
(arr: Any) =>
gen.write(arr.asInstanceOf[ArrayData])
getAndReset()
+ case MapType(_: DataType, _: StructType, _: Boolean) =>
+ (map: Any) =>
+ val mapType = child.dataType.asInstanceOf[MapType]
+ gen.write(map.asInstanceOf[MapData], mapType)
+ getAndReset()
}
}
override def dataType: DataType = StringType
override def checkInputDataTypes(): TypeCheckResult = child.dataType match {
- case _: StructType | ArrayType(_: StructType, _) =>
+ case _: StructType | ArrayType(_: StructType, _) |
+ MapType(_: DataType, _: StructType, _: Boolean) =>
try {
JacksonUtils.verifySchema(rowSchema)
TypeCheckResult.TypeCheckSuccess
--- End diff --
@HyukjinKwon yeah, you're right. I'll add some message about maptype.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81616 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81616/testReport)** for PR 18875 at commit [`b8bf38c`](https://github.com/apache/spark/commit/b8bf38cda5f3b03473f5bd1f1c4c8599e342b869).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81380/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81684 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81684/testReport)** for PR 18875 at commit [`bddf283`](https://github.com/apache/spark/commit/bddf2838868b2b676ae9eb3c595b53f56de07468).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18875
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81641 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81641/testReport)** for PR 18875 at commit [`069708c`](https://github.com/apache/spark/commit/069708c7977fe5e9239ce5c42db8337307f8a3fc).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/18875
@HyukjinKwon Should we fix the last two comments with a small follow up PR? Or in the PR of Python and R?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r138228185
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -193,14 +223,35 @@ private[sql] class JacksonGenerator(
*
* @param row The row to convert
*/
- def write(row: InternalRow): Unit = writeObject(writeFields(row, schema, rootFieldWriters))
+ def write(row: InternalRow): Unit = {
+ writeObject(writeFields(
+ fieldWriters = rootFieldWriters,
+ row = row,
+ schema = dataType.asInstanceOf[StructType]))
+ }
+
+
+ /**
+ * Transforms multiple `InternalRow`s or `MapData`s to JSON array using Jackson
+ *
+ * @param array The array of rows or maps to convert
+ */
+ def write(array: ArrayData): Unit = writeArray(writeArrayData(
+ fieldWriter = arrElementWriter,
+ array = array
+ ))
/**
- * Transforms multiple `InternalRow`s to JSON array using Jackson
+ * Transforms a single `MapData` to JSON object using Jackson
*
- * @param array The array of rows to convert
+ * @param map a map to convert
*/
- def write(array: ArrayData): Unit = writeArray(writeArrayData(array, arrElementWriter))
+ def write(map: MapData): Unit = {
+ writeObject(writeMapData(
+ fieldWriter = mapElementWriter,
--- End diff --
Let's add a small comment saying this actually triggers the proper validation.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r132615092
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala ---
@@ -186,6 +186,18 @@ class JsonFunctionsSuite extends QueryTest with SharedSQLContext {
Row("""[{"_1":1}]""") :: Nil)
}
+ test("to_json - map") {
+ val df1 = Seq(Map("a" -> Tuple1(1))).toDF("a")
+ val df2 = Seq(Map(Tuple1(1) -> Tuple1(1))).toDF("a")
+
+ checkAnswer(
+ df1.select(to_json($"a")),
+ Row("""{"a":{"_1":1}}""") :: Nil)
+ checkAnswer(
+ df2.select(to_json($"a")),
+ Row("""{"[0,1]":{"_1":1}}""") :: Nil)
--- End diff --
Actually, I'm not sure what answer is it but I got `[0,1]` using
```
scala> Seq(Tuple1(Tuple1(Map(Tuple1(1)->Tuple1(1))))).toDF("a").select(to_json($"a")).show()
+--------------------+
| structstojson(a)|
+--------------------+
|[{"_1":{"[0,1]":{...|
+--------------------+
```
so I think this answer should be correct.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #80984 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80984/testReport)** for PR 18875 at commit [`a254f89`](https://github.com/apache/spark/commit/a254f8923622d9b85aef71f1b979e7fb85260ec8).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137281077
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -22,24 +22,49 @@ import java.io.Writer
import com.fasterxml.jackson.core._
import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
import org.apache.spark.sql.catalyst.expressions.SpecializedGetters
import org.apache.spark.sql.catalyst.util.{ArrayData, DateTimeUtils, MapData}
import org.apache.spark.sql.types._
private[sql] class JacksonGenerator(
--- End diff --
We should add comment for `JacksonGenerator`. E.g.,
`JackGenerator` can only be initialized with a `StructType` or a `MapType`. Once it is initialized with
`StructType`, it can be used to write out a struct or an array of struct. Once it is initialized with `MapType`, it can be used to write out a map. An exception will be thrown if trying to write out a struct if it is initialized with a `MapType`, and vice verse.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r138087980
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/json/JacksonGeneratorSuite.scala ---
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.json
+
+import java.io.CharArrayWriter
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, DateTimeUtils, GenericArrayData}
+import org.apache.spark.sql.types._
+
+class JacksonGeneratorSuite extends SparkFunSuite {
+
+ val gmtId = DateTimeUtils.TimeZoneGMT.getID
+ val option = new JSONOptions(Map.empty, gmtId)
+
+ test("initial with StructType and write out a row") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = InternalRow(1)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """{"a":1}""")
+ }
+
+ test("initial with StructType and write out rows") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = new GenericArrayData(InternalRow(1) :: InternalRow(2) :: Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """[{"a":1},{"a":2}]""")
+ }
+
+ test("initial with StructType and write out an array with single empty row") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = new GenericArrayData(InternalRow(null) :: Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """[{}]""")
+ }
+
+ test("initial with StructType and write out an empty array") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = new GenericArrayData(Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """[]""")
+ }
+
+ test("initial with Map and write out a map data") {
+ val dataType = MapType(StringType, IntegerType)
+ val input = ArrayBasedMapData(Map("a" -> 1))
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """{"a":1}""")
+ }
+
+ test("initial with Map and write out an array of maps") {
+ val dataType = MapType(StringType, IntegerType)
+ val input = new GenericArrayData(
+ ArrayBasedMapData(Map("a" -> 1)) :: ArrayBasedMapData(Map("b" -> 2)) :: Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """[{"a":1},{"b":2}]""")
+ }
+
+ test("error handling: initial with StructType but error calling write a map") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = ArrayBasedMapData(Map("a" -> 1))
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ intercept[Exception] {
+ gen.write(input)
+ }
+ }
+
+ test("error handling: initial with StructType but error calling write an array of maps") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = new GenericArrayData(
+ ArrayBasedMapData(Map("a" -> 1)) :: ArrayBasedMapData(Map("b" -> 2)) :: Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ intercept[Exception] {
--- End diff --
This way is great! Thanks :)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r138087162
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/json/JacksonGeneratorSuite.scala ---
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.json
+
+import java.io.CharArrayWriter
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, DateTimeUtils, GenericArrayData}
+import org.apache.spark.sql.types._
+
+class JacksonGeneratorSuite extends SparkFunSuite {
+
+ val gmtId = DateTimeUtils.TimeZoneGMT.getID
+ val option = new JSONOptions(Map.empty, gmtId)
+
+ test("initial with StructType and write out a row") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = InternalRow(1)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """{"a":1}""")
+ }
+
+ test("initial with StructType and write out rows") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = new GenericArrayData(InternalRow(1) :: InternalRow(2) :: Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """[{"a":1},{"a":2}]""")
+ }
+
+ test("initial with StructType and write out an array with single empty row") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = new GenericArrayData(InternalRow(null) :: Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """[{}]""")
+ }
+
+ test("initial with StructType and write out an empty array") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = new GenericArrayData(Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """[]""")
+ }
+
+ test("initial with Map and write out a map data") {
+ val dataType = MapType(StringType, IntegerType)
+ val input = ArrayBasedMapData(Map("a" -> 1))
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """{"a":1}""")
+ }
+
+ test("initial with Map and write out an array of maps") {
+ val dataType = MapType(StringType, IntegerType)
+ val input = new GenericArrayData(
+ ArrayBasedMapData(Map("a" -> 1)) :: ArrayBasedMapData(Map("b" -> 2)) :: Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """[{"a":1},{"b":2}]""")
+ }
+
+ test("error handling: initial with StructType but error calling write a map") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = ArrayBasedMapData(Map("a" -> 1))
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ intercept[Exception] {
+ gen.write(input)
+ }
+ }
+
+ test("error handling: initial with StructType but error calling write an array of maps") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = new GenericArrayData(
+ ArrayBasedMapData(Map("a" -> 1)) :: ArrayBasedMapData(Map("b" -> 2)) :: Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ intercept[Exception] {
--- End diff --
Oh, this should be a workaround to try. Then we can get `UnsupportedOperationException`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81604/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137975716
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -193,14 +228,32 @@ private[sql] class JacksonGenerator(
*
* @param row The row to convert
*/
- def write(row: InternalRow): Unit = writeObject(writeFields(row, schema, rootFieldWriters))
+ def write(row: InternalRow): Unit = dataType match {
+ case st: StructType =>
+ writeObject(writeFields(row, st, rootFieldWriters))
+ case _ => throw new UnsupportedOperationException(
--- End diff --
ok. I'll remove it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r136997720
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -27,21 +27,45 @@ import org.apache.spark.sql.catalyst.util.{ArrayData, DateTimeUtils, MapData}
import org.apache.spark.sql.types._
private[sql] class JacksonGenerator(
- schema: StructType,
+ childType: DataType,
+ rowSchema: StructType,
--- End diff --
That's said, for now `JacksonGenerator` supports to write out a struct or an array of struct. After revising, it can also support to write out map, array of map.
Once a `JacksonGenerator` is initialized with a `MapType`, it is not valid to call the API `write(row: InternalRow)`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80399/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81590 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81590/testReport)** for PR 18875 at commit [`223acfe`](https://github.com/apache/spark/commit/223acfeef4f8461d9c0ac1b4bd3a997849d6008b).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r136996807
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -27,21 +27,45 @@ import org.apache.spark.sql.catalyst.util.{ArrayData, DateTimeUtils, MapData}
import org.apache.spark.sql.types._
private[sql] class JacksonGenerator(
- schema: StructType,
+ childType: DataType,
+ rowSchema: StructType,
--- End diff --
We can loose the type of `rowSchema` from `StructType` to general `DataType`. Since only `StructType` and `MapType` are valid options, we can check this in `JacksonGenerator`'s constructor.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81620/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #80500 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80500/testReport)** for PR 18875 at commit [`43ec2dd`](https://github.com/apache/spark/commit/43ec2dd0704d4ef7cd013ffc0185de570072d172).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on the issue:
https://github.com/apache/spark/pull/18875
@HyukjinKwon We have finished the `MapType` and `ArrayType` of `MapType`s supporting. Please take a look when you are available. Thanks :)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r132088249
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
@@ -659,13 +660,19 @@ case class StructsToJson(
(arr: Any) =>
gen.write(arr.asInstanceOf[ArrayData])
getAndReset()
+ case MapType(_: DataType, _: StructType, _: Boolean) =>
+ (map: Any) =>
+ val mapType = child.dataType.asInstanceOf[MapType]
+ gen.write(map.asInstanceOf[MapData], mapType)
+ getAndReset()
}
}
override def dataType: DataType = StringType
override def checkInputDataTypes(): TypeCheckResult = child.dataType match {
- case _: StructType | ArrayType(_: StructType, _) =>
+ case _: StructType | ArrayType(_: StructType, _) |
+ MapType(_: DataType, _: StructType, _: Boolean) =>
--- End diff --
I see. Looks now we have assumption that `JacksonGenerator` only takes `StructType` now. I think expression name should be fine and I guess we are okay to rename it if it looks required (actually I did it to `StructsToJson` :) ).
Would it be difficult to support `MapType` like `StructType`? I took a quick look and it looks quite a lot of places to fix though. Up to my understanding, we could load JSON both `MapType` and `StructType` too.
```
scala> spark.read.schema("a map<string, int>").json(Seq("""{"a": {"a": 2}}""").toDS).show()
+-----------+
| a|
+-----------+
|Map(a -> 2)|
+-----------+
scala> spark.read.schema("a struct<a: int>").json(Seq("""{"a": {"a": 2}}""").toDS).show()
+---+
| a|
+---+
|[2]|
+---+
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81566 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81566/testReport)** for PR 18875 at commit [`7942039`](https://github.com/apache/spark/commit/7942039c28e7ef9f751b2f2a3ad0ef326f6abe7c).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137048876
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -27,21 +27,45 @@ import org.apache.spark.sql.catalyst.util.{ArrayData, DateTimeUtils, MapData}
import org.apache.spark.sql.types._
private[sql] class JacksonGenerator(
- schema: StructType,
+ childType: DataType,
+ rowSchema: StructType,
--- End diff --
thanks for review =). I will follow your suggestion to change it.
But I think `JacksonGenerator` only support write out an arbitrary map, it doesn't support to write out an array of map yet. Should I need to fix it? I think that maybe an issue for supporting arbitrary array?
Should I need to do some check for API `write(row: InternalRow)` calling?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81454 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81454/testReport)** for PR 18875 at commit [`8424c90`](https://github.com/apache/spark/commit/8424c9063da056b966c7561918fadf951fa4ec5b).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r132614534
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/JsonExpressionsSuite.scala ---
@@ -590,4 +590,24 @@ class JsonExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
"""{"t":"2015-12-31T16:00:00"}"""
)
}
+
+ test("SPARK-21513: to_json support map[string, struct] to json") {
+ val schema = MapType(StringType, StructType(StructField("a", IntegerType) :: Nil))
+ val input = Literal.create(ArrayBasedMapData(Map("test" -> InternalRow(1))), schema)
+ checkEvaluation(
+ StructsToJson(Map.empty, input, gmtId),
+ """{"test":{"a":1}}"""
+ )
+ }
+
+ test("SPARK-21513: to_json support map[struct, struct] to json") {
+ val schema = MapType(StructType(StructField("a", IntegerType) :: Nil),
+ StructType(StructField("b", IntegerType) :: Nil))
+ val input = Literal.create(ArrayBasedMapData(Map(InternalRow(1) -> InternalRow(2))), schema)
+ checkEvaluation(
+ StructsToJson(Map.empty, input, gmtId),
--- End diff --
ditto.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #80399 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80399/testReport)** for PR 18875 at commit [`4cf4212`](https://github.com/apache/spark/commit/4cf42127a7c1044263a69bc7c2562f198558eb27).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137226222
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -22,24 +22,44 @@ import java.io.Writer
import com.fasterxml.jackson.core._
import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
import org.apache.spark.sql.catalyst.expressions.SpecializedGetters
import org.apache.spark.sql.catalyst.util.{ArrayData, DateTimeUtils, MapData}
import org.apache.spark.sql.types._
private[sql] class JacksonGenerator(
- schema: StructType,
+ rowSchema: DataType,
writer: Writer,
options: JSONOptions) {
+
--- End diff --
Nit: We don't need to add this extra space.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/18875
@HyukjinKwon any more comments on this change? We can support the arbitrary Map type and rename this expression in following PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r132614524
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/JsonExpressionsSuite.scala ---
@@ -590,4 +590,24 @@ class JsonExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
"""{"t":"2015-12-31T16:00:00"}"""
)
}
+
+ test("SPARK-21513: to_json support map[string, struct] to json") {
+ val schema = MapType(StringType, StructType(StructField("a", IntegerType) :: Nil))
+ val input = Literal.create(ArrayBasedMapData(Map("test" -> InternalRow(1))), schema)
+ checkEvaluation(
+ StructsToJson(Map.empty, input, gmtId),
--- End diff --
I think we don't need `gmtId` here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81182 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81182/testReport)** for PR 18875 at commit [`771e77a`](https://github.com/apache/spark/commit/771e77ab3ea23f672a668d4e171f8d1d1e956cc9).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r131936404
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
@@ -659,13 +660,19 @@ case class StructsToJson(
(arr: Any) =>
gen.write(arr.asInstanceOf[ArrayData])
getAndReset()
+ case MapType(_: DataType, _: StructType, _: Boolean) =>
+ (map: Any) =>
+ val mapType = child.dataType.asInstanceOf[MapType]
+ gen.write(map.asInstanceOf[MapData], mapType)
+ getAndReset()
}
}
override def dataType: DataType = StringType
override def checkInputDataTypes(): TypeCheckResult = child.dataType match {
- case _: StructType | ArrayType(_: StructType, _) =>
+ case _: StructType | ArrayType(_: StructType, _) |
+ MapType(_: DataType, _: StructType, _: Boolean) =>
--- End diff --
I think this depends how `MapType` is proposed to use here. If we treat a map like a struct similarly, we can support arbitrary map type. Then we need to explicitly clarify it in expression description.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137977153
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/json/JacksonGeneratorSuite.scala ---
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.json
+
+import java.io.CharArrayWriter
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, DateTimeUtils, GenericArrayData}
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+class JacksonGeneratorSuite extends SparkFunSuite {
+
+ val gmtId = DateTimeUtils.TimeZoneGMT.getID
+ val option = new JSONOptions(Map.empty, gmtId)
+ val writer = new CharArrayWriter()
+ def getAndReset(gen: JacksonGenerator): UTF8String = {
+ gen.flush()
+ val json = writer.toString
+ writer.reset()
+ UTF8String.fromString(json)
+ }
+
+ test("initial with StructType and write out a row") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = InternalRow(1)
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ assert(getAndReset(gen) === UTF8String.fromString("""{"a":1}"""))
--- End diff --
Yes, maybe we can just `writer.toString` and assert it with the normal string.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r138078894
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/JsonExpressionsSuite.scala ---
@@ -612,6 +612,54 @@ class JsonExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
)
}
+ test("SPARK-21513: to_json support map[string, struct] to json") {
+ val schema = MapType(StringType, StructType(StructField("a", IntegerType) :: Nil))
+ val input = Literal.create(ArrayBasedMapData(Map("test" -> InternalRow(1))), schema)
+ checkEvaluation(
+ StructsToJson(Map.empty, input),
+ """{"test":{"a":1}}"""
+ )
+ }
+
+ test("SPARK-21513: to_json support map[struct, struct] to json") {
+ val schema = MapType(StructType(StructField("a", IntegerType) :: Nil),
+ StructType(StructField("b", IntegerType) :: Nil))
+ val input = Literal.create(ArrayBasedMapData(Map(InternalRow(1) -> InternalRow(2))), schema)
+ checkEvaluation(
+ StructsToJson(Map.empty, input),
+ """{"[1]":{"b":2}}"""
+ )
+ }
+
+ test("SPARK-21513: to_json support map[string, integer] to json") {
+ val schema = MapType(StringType, IntegerType)
+ val input = Literal.create(ArrayBasedMapData(Map("a" -> 1)), schema)
+ checkEvaluation(
+ StructsToJson(Map.empty, input),
+ """{"a":1}"""
+ )
+ }
+
+ test("to_json - array with maps") {
+ val inputSchema = ArrayType(MapType(StringType, IntegerType))
+ val input = new GenericArrayData(ArrayBasedMapData(
+ Map("a" -> 1)) :: ArrayBasedMapData(Map("b" -> 2)) :: Nil)
+ val output = """[{"a":1},{"b":2}]"""
+ checkEvaluation(
+ StructsToJson(Map.empty, Literal.create(input, inputSchema), gmtId),
+ output)
+ }
+
+ test("to_json - array with single map") {
+ val inputSchema = ArrayType(MapType(StringType, IntegerType))
+ val input = new GenericArrayData(ArrayBasedMapData(
+ Map("a" -> 1)) :: Nil)
--- End diff --
Let's make this inlined.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18875
@viirya and @goldmedal, I am sorry. I misread https://github.com/apache/spark/pull/18875#discussion_r132613913. I was thinking arbitrary map support here too because I thought arbitrary map support should be much more common in practice. Could we do this arbitrary map support too if it won't be too difficult? I was thinking array / renaming could be done in a followup.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137977281
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -193,14 +228,32 @@ private[sql] class JacksonGenerator(
*
* @param row The row to convert
*/
- def write(row: InternalRow): Unit = writeObject(writeFields(row, schema, rootFieldWriters))
+ def write(row: InternalRow): Unit = dataType match {
+ case st: StructType =>
+ writeObject(writeFields(row, st, rootFieldWriters))
+ case _ => throw new UnsupportedOperationException(
--- End diff --
Do you mean `dataType` now is a map type but `writeFields`'s 2nd parameter is a `StructType`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18875
I think including them in the PR of R and Python would be nicer.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r131904051
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
@@ -659,13 +660,19 @@ case class StructsToJson(
(arr: Any) =>
gen.write(arr.asInstanceOf[ArrayData])
getAndReset()
+ case MapType(_: DataType, _: StructType, _: Boolean) =>
+ (map: Any) =>
+ val mapType = child.dataType.asInstanceOf[MapType]
+ gen.write(map.asInstanceOf[MapData], mapType)
+ getAndReset()
}
}
override def dataType: DataType = StringType
override def checkInputDataTypes(): TypeCheckResult = child.dataType match {
- case _: StructType | ArrayType(_: StructType, _) =>
+ case _: StructType | ArrayType(_: StructType, _) |
+ MapType(_: DataType, _: StructType, _: Boolean) =>
try {
JacksonUtils.verifySchema(rowSchema)
TypeCheckResult.TypeCheckSuccess
--- End diff --
I guess we need to fix the exception message in 684L.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r131946027
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
@@ -659,13 +660,19 @@ case class StructsToJson(
(arr: Any) =>
gen.write(arr.asInstanceOf[ArrayData])
getAndReset()
+ case MapType(_: DataType, _: StructType, _: Boolean) =>
+ (map: Any) =>
+ val mapType = child.dataType.asInstanceOf[MapType]
+ gen.write(map.asInstanceOf[MapData], mapType)
+ getAndReset()
}
}
override def dataType: DataType = StringType
override def checkInputDataTypes(): TypeCheckResult = child.dataType match {
- case _: StructType | ArrayType(_: StructType, _) =>
+ case _: StructType | ArrayType(_: StructType, _) |
+ MapType(_: DataType, _: StructType, _: Boolean) =>
--- End diff --
yeah, just like @viirya said. This expression is named `StructsToJson`.
Another reason is that it use `JacksonGenerator` which will need a `rowSchema` when initializing to generate JSON string. Therefore, it must be restricted to `StructType` here.
or can we create another expression to support arbitrary map type?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137226892
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -22,24 +22,44 @@ import java.io.Writer
import com.fasterxml.jackson.core._
import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
import org.apache.spark.sql.catalyst.expressions.SpecializedGetters
import org.apache.spark.sql.catalyst.util.{ArrayData, DateTimeUtils, MapData}
import org.apache.spark.sql.types._
private[sql] class JacksonGenerator(
- schema: StructType,
+ rowSchema: DataType,
writer: Writer,
options: JSONOptions) {
+
// A `ValueWriter` is responsible for writing a field of an `InternalRow` to appropriate
// JSON data. Here we are using `SpecializedGetters` rather than `InternalRow` so that
// we can directly access data in `ArrayData` without the help of `SpecificMutableRow`.
private type ValueWriter = (SpecializedGetters, Int) => Unit
+ // `JackGenerator` only supports to write out a struct, an array of struct or an arbitrary map
+ rowSchema match {
+ case _: StructType | _: MapType =>
+ TypeCheckResult.TypeCheckSuccess
+ case _ => TypeCheckResult.TypeCheckFailure(
+ s"Input type ${rowSchema.simpleString} must be a struct or a map")
--- End diff --
`TypeCheckResult` actually doesn't prevent you to construct this `JacksonGenerator`. We can throw an `UnsupportedOperationException` for unsupported types.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81414 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81414/testReport)** for PR 18875 at commit [`40796bd`](https://github.com/apache/spark/commit/40796bd0cc22f300946bc27e0ffae6c104b5167d).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81632/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81630 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81630/testReport)** for PR 18875 at commit [`b71b56a`](https://github.com/apache/spark/commit/b71b56ab49ba048f397459a18f946422d9440b29).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r138110523
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/json/JacksonGeneratorSuite.scala ---
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.json
+
+import java.io.CharArrayWriter
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, DateTimeUtils, GenericArrayData}
+import org.apache.spark.sql.types._
+
+class JacksonGeneratorSuite extends SparkFunSuite {
+
+ val gmtId = DateTimeUtils.TimeZoneGMT.getID
+ val option = new JSONOptions(Map.empty, gmtId)
+
+ test("initial with StructType and write out a row") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = InternalRow(1)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """{"a":1}""")
+ }
+
+ test("initial with StructType and write out rows") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = new GenericArrayData(InternalRow(1) :: InternalRow(2) :: Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """[{"a":1},{"a":2}]""")
+ }
+
+ test("initial with StructType and write out an array with single empty row") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = new GenericArrayData(InternalRow(null) :: Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """[{}]""")
+ }
+
+ test("initial with StructType and write out an empty array") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = new GenericArrayData(Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """[]""")
+ }
+
+ test("initial with Map and write out a map data") {
+ val dataType = MapType(StringType, IntegerType)
+ val input = ArrayBasedMapData(Map("a" -> 1))
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """{"a":1}""")
+ }
+
+ test("initial with Map and write out an array of maps") {
+ val dataType = MapType(StringType, IntegerType)
+ val input = new GenericArrayData(
+ ArrayBasedMapData(Map("a" -> 1)) :: ArrayBasedMapData(Map("b" -> 2)) :: Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """[{"a":1},{"b":2}]""")
+ }
+
+ test("error handling: initial with StructType but error calling write a map") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = ArrayBasedMapData(Map("a" -> 1))
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ intercept[Exception] {
+ gen.write(input)
+ }
+ }
+
+ test("error handling: initial with StructType but error calling write an array of maps") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = new GenericArrayData(
+ ArrayBasedMapData(Map("a" -> 1)) :: ArrayBasedMapData(Map("b" -> 2)) :: Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ intercept[Exception] {
--- End diff --
ok. I'll remove them.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r131903650
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
@@ -659,13 +660,19 @@ case class StructsToJson(
(arr: Any) =>
gen.write(arr.asInstanceOf[ArrayData])
getAndReset()
+ case MapType(_: DataType, _: StructType, _: Boolean) =>
+ (map: Any) =>
+ val mapType = child.dataType.asInstanceOf[MapType]
+ gen.write(map.asInstanceOf[MapData], mapType)
+ getAndReset()
}
}
override def dataType: DataType = StringType
override def checkInputDataTypes(): TypeCheckResult = child.dataType match {
- case _: StructType | ArrayType(_: StructType, _) =>
+ case _: StructType | ArrayType(_: StructType, _) |
+ MapType(_: DataType, _: StructType, _: Boolean) =>
--- End diff --
Hm.. is there any reason to allow `map<.., struct>` only?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137226390
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -22,24 +22,44 @@ import java.io.Writer
import com.fasterxml.jackson.core._
import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
import org.apache.spark.sql.catalyst.expressions.SpecializedGetters
import org.apache.spark.sql.catalyst.util.{ArrayData, DateTimeUtils, MapData}
import org.apache.spark.sql.types._
private[sql] class JacksonGenerator(
- schema: StructType,
+ rowSchema: DataType,
--- End diff --
Because it now can be a map type and not a schema, may be just name it as `dataType: DataType`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81611/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81589 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81589/testReport)** for PR 18875 at commit [`8aa4307`](https://github.com/apache/spark/commit/8aa430749357f9e393b9b5856dbee7425af77302).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on the issue:
https://github.com/apache/spark/pull/18875
@HyukjinKwon Thanks for the reference! I'll check it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18875
@goldmedal, before proceeding the review, assuming from the PR description, I guess your PR targets SQL and Scala APIs for now, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81455/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137710147
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -26,20 +26,50 @@ import org.apache.spark.sql.catalyst.expressions.SpecializedGetters
import org.apache.spark.sql.catalyst.util.{ArrayData, DateTimeUtils, MapData}
import org.apache.spark.sql.types._
+// `JackGenerator` can only be initialized with a `StructType` or a `MapType`.
+// Once it is initialized with `StructType`, it can be used to write out a struct or an array of
+// struct. Once it is initialized with `MapType`, it can be used to write out a map. An exception
+// will be thrown if trying to write out a struct if it is initialized with a `MapType`,
+// and vice verse.
--- End diff --
ok. I'll modify it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137987843
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -193,14 +228,32 @@ private[sql] class JacksonGenerator(
*
* @param row The row to convert
*/
- def write(row: InternalRow): Unit = writeObject(writeFields(row, schema, rootFieldWriters))
+ def write(row: InternalRow): Unit = dataType match {
+ case st: StructType =>
+ writeObject(writeFields(row, st, rootFieldWriters))
+ case _ => throw new UnsupportedOperationException(
--- End diff --
It will throw `ClassCastException`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81566/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137977826
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/json/JacksonGeneratorSuite.scala ---
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.json
+
+import java.io.CharArrayWriter
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, DateTimeUtils, GenericArrayData}
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+class JacksonGeneratorSuite extends SparkFunSuite {
+
+ val gmtId = DateTimeUtils.TimeZoneGMT.getID
+ val option = new JSONOptions(Map.empty, gmtId)
+ val writer = new CharArrayWriter()
+ def getAndReset(gen: JacksonGenerator): UTF8String = {
+ gen.flush()
+ val json = writer.toString
+ writer.reset()
+ UTF8String.fromString(json)
+ }
+
+ test("initial with StructType and write out a row") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = InternalRow(1)
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ assert(getAndReset(gen) === UTF8String.fromString("""{"a":1}"""))
--- End diff --
Yeah, but it's just two lines. Depends on you to have a function or not. At least it shouldn't be called `getAndReset` now.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137950153
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -193,14 +228,58 @@ private[sql] class JacksonGenerator(
*
* @param row The row to convert
*/
- def write(row: InternalRow): Unit = writeObject(writeFields(row, schema, rootFieldWriters))
+ def write(row: InternalRow): Unit = dataType match {
+ case st: StructType =>
+ writeObject(writeFields(row, st, rootFieldWriters))
+ case _ => throw new UnsupportedOperationException(
+ s"`JacksonGenerator` can only be used to write out a row when initialized with `StructType`.")
+ }
/**
- * Transforms multiple `InternalRow`s to JSON array using Jackson
+ * Transforms multiple `InternalRow`s or `MapData`s to JSON array using Jackson
*
- * @param array The array of rows to convert
+ * @param array The array of rows or maps to convert
*/
- def write(array: ArrayData): Unit = writeArray(writeArrayData(array, arrElementWriter))
+ def write(array: ArrayData): Unit = dataType match {
+ case st: StructType =>
+ try {
--- End diff --
If we assume that the caller know what it's doing, we can get rid of this check.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on the issue:
https://github.com/apache/spark/pull/18875
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137977643
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/json/JacksonGeneratorSuite.scala ---
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.json
+
+import java.io.CharArrayWriter
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, DateTimeUtils, GenericArrayData}
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+class JacksonGeneratorSuite extends SparkFunSuite {
+
+ val gmtId = DateTimeUtils.TimeZoneGMT.getID
+ val option = new JSONOptions(Map.empty, gmtId)
+ val writer = new CharArrayWriter()
+ def getAndReset(gen: JacksonGenerator): UTF8String = {
+ gen.flush()
+ val json = writer.toString
+ writer.reset()
+ UTF8String.fromString(json)
+ }
+
+ test("initial with StructType and write out a row") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = InternalRow(1)
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ assert(getAndReset(gen) === UTF8String.fromString("""{"a":1}"""))
--- End diff --
@viirya I think we also need to do `gen.flush()`, right? So maybe we can keep `getAndReset` and modify it as below. Will it be better?
```
def getAndReset(gen: JacksonGenerator, writer: Writer): String = {
gen.flush()
writer.toString
}
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18875
yea this looks in the right direction and I guess a good reference should be https://github.com/goldmedal/spark/commit/0cdcf9114527a2c359c25e46fd6556b3855bfb28
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137227713
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -22,24 +22,44 @@ import java.io.Writer
import com.fasterxml.jackson.core._
import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
import org.apache.spark.sql.catalyst.expressions.SpecializedGetters
import org.apache.spark.sql.catalyst.util.{ArrayData, DateTimeUtils, MapData}
import org.apache.spark.sql.types._
private[sql] class JacksonGenerator(
- schema: StructType,
+ rowSchema: DataType,
writer: Writer,
options: JSONOptions) {
+
// A `ValueWriter` is responsible for writing a field of an `InternalRow` to appropriate
// JSON data. Here we are using `SpecializedGetters` rather than `InternalRow` so that
// we can directly access data in `ArrayData` without the help of `SpecificMutableRow`.
private type ValueWriter = (SpecializedGetters, Int) => Unit
+ // `JackGenerator` only supports to write out a struct, an array of struct or an arbitrary map
+ rowSchema match {
+ case _: StructType | _: MapType =>
+ TypeCheckResult.TypeCheckSuccess
+ case _ => TypeCheckResult.TypeCheckFailure(
+ s"Input type ${rowSchema.simpleString} must be a struct or a map")
+ }
+
// `ValueWriter`s for all fields of the schema
- private val rootFieldWriters: Array[ValueWriter] = schema.map(_.dataType).map(makeWriter).toArray
+ private lazy val rootFieldWriters: Array[ValueWriter] = {
+ rowSchema.asInstanceOf[StructType].map(_.dataType).map(makeWriter).toArray
+ }
+
// `ValueWriter` for array data storing rows of the schema.
- private val arrElementWriter: ValueWriter = (arr: SpecializedGetters, i: Int) => {
- writeObject(writeFields(arr.getStruct(i, schema.length), schema, rootFieldWriters))
+ private lazy val arrElementWriter: ValueWriter = {
+ (arr: SpecializedGetters, i: Int) => {
+ val schema: StructType = rowSchema.asInstanceOf[StructType]
+ writeObject(writeFields(arr.getStruct(i, schema.length), schema, rootFieldWriters))
+ }
+ }
+
+ private lazy val mapElementWriter: ValueWriter = {
+ makeWriter(rowSchema.asInstanceOf[MapType].valueType)
--- End diff --
ditto.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81537/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81590 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81590/testReport)** for PR 18875 at commit [`223acfe`](https://github.com/apache/spark/commit/223acfeef4f8461d9c0ac1b4bd3a997849d6008b).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137227541
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -22,24 +22,44 @@ import java.io.Writer
import com.fasterxml.jackson.core._
import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
import org.apache.spark.sql.catalyst.expressions.SpecializedGetters
import org.apache.spark.sql.catalyst.util.{ArrayData, DateTimeUtils, MapData}
import org.apache.spark.sql.types._
private[sql] class JacksonGenerator(
- schema: StructType,
+ rowSchema: DataType,
writer: Writer,
options: JSONOptions) {
+
// A `ValueWriter` is responsible for writing a field of an `InternalRow` to appropriate
// JSON data. Here we are using `SpecializedGetters` rather than `InternalRow` so that
// we can directly access data in `ArrayData` without the help of `SpecificMutableRow`.
private type ValueWriter = (SpecializedGetters, Int) => Unit
+ // `JackGenerator` only supports to write out a struct, an array of struct or an arbitrary map
+ rowSchema match {
+ case _: StructType | _: MapType =>
+ TypeCheckResult.TypeCheckSuccess
+ case _ => TypeCheckResult.TypeCheckFailure(
+ s"Input type ${rowSchema.simpleString} must be a struct or a map")
+ }
+
// `ValueWriter`s for all fields of the schema
- private val rootFieldWriters: Array[ValueWriter] = schema.map(_.dataType).map(makeWriter).toArray
+ private lazy val rootFieldWriters: Array[ValueWriter] = {
+ rowSchema.asInstanceOf[StructType].map(_.dataType).map(makeWriter).toArray
--- End diff --
We should take care when accessing `rootFieldWriters` if the given data type is a map type. E.g.:
private lazy val rootFieldWriters: Array[ValueWriter] = dataType match {
case st: StructType => st.map(_.dataType).map(makeWriter).toArray
case mt: MapType => throw UnsupportedOperationException(".......")
}
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r138073822
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/json/JacksonGeneratorSuite.scala ---
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.json
+
+import java.io.CharArrayWriter
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, DateTimeUtils, GenericArrayData}
+import org.apache.spark.sql.types._
+
+class JacksonGeneratorSuite extends SparkFunSuite {
+
+ val gmtId = DateTimeUtils.TimeZoneGMT.getID
+ val option = new JSONOptions(Map.empty, gmtId)
+
+ test("initial with StructType and write out a row") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = InternalRow(1)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """{"a":1}""")
+ }
+
+ test("initial with StructType and write out rows") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = new GenericArrayData(InternalRow(1) :: InternalRow(2) :: Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """[{"a":1},{"a":2}]""")
+ }
+
+ test("initial with StructType and write out an array with single empty row") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = new GenericArrayData(InternalRow(null) :: Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """[{}]""")
+ }
+
+ test("initial with StructType and write out an empty array") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = new GenericArrayData(Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """[]""")
+ }
+
+ test("initial with Map and write out a map data") {
+ val dataType = MapType(StringType, IntegerType)
+ val input = ArrayBasedMapData(Map("a" -> 1))
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """{"a":1}""")
+ }
+
+ test("initial with Map and write out an array of maps") {
+ val dataType = MapType(StringType, IntegerType)
+ val input = new GenericArrayData(
+ ArrayBasedMapData(Map("a" -> 1)) :: ArrayBasedMapData(Map("b" -> 2)) :: Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """[{"a":1},{"b":2}]""")
+ }
+
+ test("error handling: initial with StructType but error calling write a map") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = ArrayBasedMapData(Map("a" -> 1))
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ intercept[Exception] {
+ gen.write(input)
+ }
+ }
+
+ test("error handling: initial with StructType but error calling write an array of maps") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = new GenericArrayData(
+ ArrayBasedMapData(Map("a" -> 1)) :: ArrayBasedMapData(Map("b" -> 2)) :: Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ intercept[Exception] {
--- End diff --
This looks throwing `ClassCastException` rather than `UnsupportedOperationException` as I guess intended.
One workaround I could think now is, calling it by named arguments, for example:
```diff
- writeObject(writeMapData(map, dataType.asInstanceOf[MapType]), mapElementWriter)
+ writeObject(writeMapData(
+ fieldWriter = mapElementWriter,
+ map = map,
+ mapType = dataType.asInstanceOf[MapType]))
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r138107641
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/json/JacksonGeneratorSuite.scala ---
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.json
+
+import java.io.CharArrayWriter
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, DateTimeUtils, GenericArrayData}
+import org.apache.spark.sql.types._
+
+class JacksonGeneratorSuite extends SparkFunSuite {
+
+ val gmtId = DateTimeUtils.TimeZoneGMT.getID
+ val option = new JSONOptions(Map.empty, gmtId)
+
+ test("initial with StructType and write out a row") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = InternalRow(1)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """{"a":1}""")
+ }
+
+ test("initial with StructType and write out rows") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = new GenericArrayData(InternalRow(1) :: InternalRow(2) :: Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """[{"a":1},{"a":2}]""")
+ }
+
+ test("initial with StructType and write out an array with single empty row") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = new GenericArrayData(InternalRow(null) :: Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """[{}]""")
+ }
+
+ test("initial with StructType and write out an empty array") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = new GenericArrayData(Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """[]""")
+ }
+
+ test("initial with Map and write out a map data") {
+ val dataType = MapType(StringType, IntegerType)
+ val input = ArrayBasedMapData(Map("a" -> 1))
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """{"a":1}""")
+ }
+
+ test("initial with Map and write out an array of maps") {
+ val dataType = MapType(StringType, IntegerType)
+ val input = new GenericArrayData(
+ ArrayBasedMapData(Map("a" -> 1)) :: ArrayBasedMapData(Map("b" -> 2)) :: Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """[{"a":1},{"b":2}]""")
+ }
+
+ test("error handling: initial with StructType but error calling write a map") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = ArrayBasedMapData(Map("a" -> 1))
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ intercept[Exception] {
+ gen.write(input)
+ }
+ }
+
+ test("error handling: initial with StructType but error calling write an array of maps") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = new GenericArrayData(
+ ArrayBasedMapData(Map("a" -> 1)) :: ArrayBasedMapData(Map("b" -> 2)) :: Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ intercept[Exception] {
--- End diff --
I see. I think this array type validation was already failed though. Let's remove out the test cases related with `ClassCastException` too and let's do this in a separate PR. I think this does not block this PR as it is not a regression.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r138094337
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/json/JacksonGeneratorSuite.scala ---
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.json
+
+import java.io.CharArrayWriter
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, DateTimeUtils, GenericArrayData}
+import org.apache.spark.sql.types._
+
+class JacksonGeneratorSuite extends SparkFunSuite {
+
+ val gmtId = DateTimeUtils.TimeZoneGMT.getID
+ val option = new JSONOptions(Map.empty, gmtId)
+
+ test("initial with StructType and write out a row") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = InternalRow(1)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """{"a":1}""")
+ }
+
+ test("initial with StructType and write out rows") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = new GenericArrayData(InternalRow(1) :: InternalRow(2) :: Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """[{"a":1},{"a":2}]""")
+ }
+
+ test("initial with StructType and write out an array with single empty row") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = new GenericArrayData(InternalRow(null) :: Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """[{}]""")
+ }
+
+ test("initial with StructType and write out an empty array") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = new GenericArrayData(Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """[]""")
+ }
+
+ test("initial with Map and write out a map data") {
+ val dataType = MapType(StringType, IntegerType)
+ val input = ArrayBasedMapData(Map("a" -> 1))
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """{"a":1}""")
+ }
+
+ test("initial with Map and write out an array of maps") {
+ val dataType = MapType(StringType, IntegerType)
+ val input = new GenericArrayData(
+ ArrayBasedMapData(Map("a" -> 1)) :: ArrayBasedMapData(Map("b" -> 2)) :: Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """[{"a":1},{"b":2}]""")
+ }
+
+ test("error handling: initial with StructType but error calling write a map") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = ArrayBasedMapData(Map("a" -> 1))
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ intercept[Exception] {
+ gen.write(input)
+ }
+ }
+
+ test("error handling: initial with StructType but error calling write an array of maps") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = new GenericArrayData(
+ ArrayBasedMapData(Map("a" -> 1)) :: ArrayBasedMapData(Map("b" -> 2)) :: Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ intercept[Exception] {
--- End diff --
@HyukjinKwon @viirya
umm... I still have problems of `ArrayType[MapType]` and `ArrayType[StructType]`. We are so hard to check which type is the elements of `ArrayData` that we can't throw `UnsupportedOperationException`. If we need to catch a narrower exception, maybe we can just catch `ClassCastException`?
Just like `interecept[ClassCastException] {`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137273266
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -22,24 +22,50 @@ import java.io.Writer
import com.fasterxml.jackson.core._
import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
import org.apache.spark.sql.catalyst.expressions.SpecializedGetters
import org.apache.spark.sql.catalyst.util.{ArrayData, DateTimeUtils, MapData}
import org.apache.spark.sql.types._
private[sql] class JacksonGenerator(
- schema: StructType,
+ dataType: DataType,
writer: Writer,
options: JSONOptions) {
// A `ValueWriter` is responsible for writing a field of an `InternalRow` to appropriate
// JSON data. Here we are using `SpecializedGetters` rather than `InternalRow` so that
// we can directly access data in `ArrayData` without the help of `SpecificMutableRow`.
private type ValueWriter = (SpecializedGetters, Int) => Unit
+ // `JackGenerator` only supports to write out a struct, an array of struct or an arbitrary map
+ dataType match {
+ case _: StructType | _: MapType =>
+ TypeCheckResult.TypeCheckSuccess
--- End diff --
ok
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137224595
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
@@ -661,6 +666,10 @@ case class StructsToJson(
(arr: Any) =>
gen.write(arr.asInstanceOf[ArrayData])
getAndReset()
+ case MapType(_: DataType, _: DataType, _: Boolean) =>
--- End diff --
For arbitrary map, we can simply write `case _: MapType => `
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81263 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81263/testReport)** for PR 18875 at commit [`960fa3f`](https://github.com/apache/spark/commit/960fa3f58fd349b05e0be7c911488ea613fc2427).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81382/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #80586 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80586/testReport)** for PR 18875 at commit [`d2a6d5e`](https://github.com/apache/spark/commit/d2a6d5e520cc6de36624b2f6dc7b730ee3b2eb8b).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #80850 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80850/testReport)** for PR 18875 at commit [`f49adcf`](https://github.com/apache/spark/commit/f49adcf1655f67c937dcb1cbe3dfc0763eb5e023).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class FeatureHasher(@Since(\"2.3.0\") override val uid: String) extends Transformer`
* `sealed abstract class SummaryBuilder `
* `case class UnresolvedCatalogRelation(tableMeta: CatalogTable) extends LeafNode `
* `case class HiveTableRelation(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r131923383
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -202,5 +202,9 @@ private[sql] class JacksonGenerator(
*/
def write(array: ArrayData): Unit = writeArray(writeArrayData(array, arrElementWriter))
+ def write(map: MapData, mapType: MapType): Unit = {
+ writeObject(writeMapData(map, mapType, makeWriter(mapType.valueType)))
--- End diff --
I think we could avoid making the writer by `makeWriter(mapType.valueType))` per each map write, by creating this once when this `JacksonGenerator`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r134469528
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -202,5 +203,9 @@ private[sql] class JacksonGenerator(
*/
def write(array: ArrayData): Unit = writeArray(writeArrayData(array, arrElementWriter))
+ def write(map: MapData, mapType: MapType): Unit = {
--- End diff --
sorry to reply too late.
I thought that others `write` function don't take any type because `JacksonGenerator` has a `schema` member. Because `writeMapData` needs `mapType`, I made `write` function of `MapType` take it.
I try to fix this problem by making `childType` matching move to `JacksonGenerator` in new commits a254f89 but I'm not sure if this way is better.
@HyukjinKwon @viirya How do you thinks about this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81643 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81643/testReport)** for PR 18875 at commit [`0f2dd18`](https://github.com/apache/spark/commit/0f2dd186b48b2b46ed1367ebae811b5767ef8598).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80585/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/18875
Thanks for @HyukjinKwon detailed review.
@goldmedal just one thing to be more specially, once you add new test into `sql/core/src/test/resources/sql-tests/inputs/json-functions.sql`. You can re-generate the golden file by running:
SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/test-only *SQLQueryTestSuite" -- -z json-functions.sql
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81546 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81546/testReport)** for PR 18875 at commit [`1df28ec`](https://github.com/apache/spark/commit/1df28ec200fd46a001b0fea9597f8b9659ea94f4).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class JacksonGeneratorSuite extends SparkFunSuite `
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81588/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/18875
@HyukjinKwon Yeah, we focus on SQL and Scala APIs for now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137227883
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -193,14 +213,27 @@ private[sql] class JacksonGenerator(
*
* @param row The row to convert
*/
- def write(row: InternalRow): Unit = writeObject(writeFields(row, schema, rootFieldWriters))
+ def write(row: InternalRow): Unit = {
+ writeObject(writeFields(row, rowSchema.asInstanceOf[StructType], rootFieldWriters))
--- End diff --
We should match `dataType` here too.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137288735
--- Diff: sql/core/src/test/resources/sql-tests/results/cross-join.sql.out ---
@@ -128,6 +128,7 @@ two 2 two 2 one 1 two 2
two 2 two 2 three 3 two 2
two 2 two 2 two 2 two 2
+
--- End diff --
ummm. That's so weird. I'm not sure about it but maybe this while line was be generated when I regenerated the golden file. I'll recover it. =P
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81461 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81461/testReport)** for PR 18875 at commit [`b01369d`](https://github.com/apache/spark/commit/b01369d300e60d710935c042bfe94451e6c06ad1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81033 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81033/testReport)** for PR 18875 at commit [`a254f89`](https://github.com/apache/spark/commit/a254f8923622d9b85aef71f1b979e7fb85260ec8).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137282296
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -193,14 +218,34 @@ private[sql] class JacksonGenerator(
*
* @param row The row to convert
*/
- def write(row: InternalRow): Unit = writeObject(writeFields(row, schema, rootFieldWriters))
+ def write(row: InternalRow): Unit = dataType match {
+ case st: StructType =>
+ writeObject(writeFields(row, st, rootFieldWriters))
+ case _ => throw new UnsupportedOperationException(
+ s"this api is only used when `JacksonGenerator` is initialized with `StructType`")
--- End diff --
`JacksonGenerator` can only be used to write out a row when initialized with `StructType`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81684 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81684/testReport)** for PR 18875 at commit [`bddf283`](https://github.com/apache/spark/commit/bddf2838868b2b676ae9eb3c595b53f56de07468).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r131951764
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -202,5 +202,9 @@ private[sql] class JacksonGenerator(
*/
def write(array: ArrayData): Unit = writeArray(writeArrayData(array, arrElementWriter))
+ def write(map: MapData, mapType: MapType): Unit = {
+ writeObject(writeMapData(map, mapType, makeWriter(mapType.valueType)))
--- End diff --
@viirya ok, got it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81416 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81416/testReport)** for PR 18875 at commit [`24a7a72`](https://github.com/apache/spark/commit/24a7a726ae613a72ee746db56e53ccc48b8038b5).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81537 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81537/testReport)** for PR 18875 at commit [`36ce961`](https://github.com/apache/spark/commit/36ce9614c078c9c0aca62a672948d8581b43e2ea).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81455 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81455/testReport)** for PR 18875 at commit [`a36309a`](https://github.com/apache/spark/commit/a36309adaaebee74ac09ac24e535c14d38fed263).
* This patch **fails SparkR unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137973919
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -26,20 +26,55 @@ import org.apache.spark.sql.catalyst.expressions.SpecializedGetters
import org.apache.spark.sql.catalyst.util.{ArrayData, DateTimeUtils, MapData}
import org.apache.spark.sql.types._
+/**
+ * `JackGenerator` can only be initialized with a `StructType` or a `MapType`.
+ * Once it is initialized with `StructType`, it can be used to write out a struct or an array of
+ * struct. Once it is initialized with `MapType`, it can be used to write out a map. An exception
--- End diff --
`Once it is initialized with ``MapType``, it can be used to write out a map or an array of map.`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137978375
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
@@ -677,14 +696,27 @@ case class StructsToJson(
override def checkInputDataTypes(): TypeCheckResult = child.dataType match {
case _: StructType | ArrayType(_: StructType, _) =>
try {
- JacksonUtils.verifySchema(rowSchema)
+ JacksonUtils.verifySchema(rowSchema.asInstanceOf[StructType])
+ TypeCheckResult.TypeCheckSuccess
+ } catch {
+ case e: UnsupportedOperationException =>
+ TypeCheckResult.TypeCheckFailure(e.getMessage)
+ }
+ case ArrayType(_: MapType, _) =>
+ TypeCheckResult.TypeCheckSuccess
+ case MapType(_: DataType, st: StructType, _: Boolean) =>
+ try {
+ JacksonUtils.verifySchema(st)
TypeCheckResult.TypeCheckSuccess
} catch {
case e: UnsupportedOperationException =>
TypeCheckResult.TypeCheckFailure(e.getMessage)
}
+ case _: MapType =>
--- End diff --
Maybe we can just do something like:
case mt: MapType =>
val st = StructType(StructField("a", mt) :: Nil)
JacksonUtils.verifySchema(st)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81631 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81631/testReport)** for PR 18875 at commit [`be62e51`](https://github.com/apache/spark/commit/be62e51109c8a438fed8d787fb2986aeb1974f82).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81033/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #80596 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80596/testReport)** for PR 18875 at commit [`c64d9c4`](https://github.com/apache/spark/commit/c64d9c49c5f42b7a72b1570f00daa34a2843be1d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80596/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81631/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137283687
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -22,24 +22,49 @@ import java.io.Writer
import com.fasterxml.jackson.core._
import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
import org.apache.spark.sql.catalyst.expressions.SpecializedGetters
import org.apache.spark.sql.catalyst.util.{ArrayData, DateTimeUtils, MapData}
import org.apache.spark.sql.types._
private[sql] class JacksonGenerator(
--- End diff --
The change to `JacksonGenerator` needs tests. Looks like we don't have existing tests for `JacksonGenerator` too. I think we are better to have one.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r138104414
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala ---
@@ -180,10 +180,30 @@ class JsonFunctionsSuite extends QueryTest with SharedSQLContext {
test("to_json - array") {
val df = Seq(Tuple1(Tuple1(1) :: Nil)).toDF("a")
+ val df2 = Seq(Tuple1(Map("a" -> 1) :: Nil)).toDF("a")
checkAnswer(
df.select(to_json($"a")),
Row("""[{"_1":1}]""") :: Nil)
+ checkAnswer(
+ df2.select(to_json($"a")),
+ Row("""[{"a":1}]""") :: Nil)
+ }
+
+ test("to_json - map") {
+ val df1 = Seq(Map("a" -> Tuple1(1))).toDF("a")
+ val df2 = Seq(Map(Tuple1(1) -> Tuple1(1))).toDF("a")
+ val df3 = Seq(Map("a" -> 1)).toDF("a")
+
+ checkAnswer(
+ df1.select(to_json($"a")),
+ Row("""{"a":{"_1":1}}""") :: Nil)
+ checkAnswer(
+ df2.select(to_json($"a")),
+ Row("""{"[0,1]":{"_1":1}}""") :: Nil)
--- End diff --
Yeah, I agree to remove it for now.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137988061
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/json/JacksonGeneratorSuite.scala ---
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.json
+
+import java.io.CharArrayWriter
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, DateTimeUtils, GenericArrayData}
+import org.apache.spark.sql.types._
+
+class JacksonGeneratorSuite extends SparkFunSuite {
+
+ val gmtId = DateTimeUtils.TimeZoneGMT.getID
+ val option = new JSONOptions(Map.empty, gmtId)
+
+ test("initial with StructType and write out a row") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = InternalRow(1)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """{"a":1}""")
+ }
+
+ test("initial with StructType and write out rows") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = new GenericArrayData(InternalRow(1) :: InternalRow(2) :: Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """[{"a":1},{"a":2}]""")
+ }
+
+ test("initial with StructType and write out an array with single empty row") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = new GenericArrayData(InternalRow(null) :: Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """[{}]""")
+ }
+
+ test("initial with StructType and write out an empty array") {
+ val dataType = StructType(StructField("a", IntegerType) :: Nil)
+ val input = new GenericArrayData(Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """[]""")
+ }
+
+ test("initial with Map and write out a map data") {
+ val dataType = MapType(StringType, IntegerType)
+ val input = ArrayBasedMapData(Map("a" -> 1))
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """{"a":1}""")
+ }
+
+ test("initial with Map and write out an array of maps") {
+ val dataType = MapType(StringType, IntegerType)
+ val input = new GenericArrayData(
+ ArrayBasedMapData(Map("a" -> 1)) :: ArrayBasedMapData(Map("b" -> 2)) :: Nil)
+ val writer = new CharArrayWriter()
+ val gen = new JacksonGenerator(dataType, writer, option)
+ gen.write(input)
+ gen.flush()
+ assert(writer.toString === """[{"a":1},{"b":2}]""")
+ }
--- End diff --
We need negative test cases, i.e, `JacksonGenerator` initialized with `StructType` but used to write map, array of map. Also `JacksonGenerator` initialized with `MapType` but used to write struct, array of struct.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137950158
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -193,14 +228,58 @@ private[sql] class JacksonGenerator(
*
* @param row The row to convert
*/
- def write(row: InternalRow): Unit = writeObject(writeFields(row, schema, rootFieldWriters))
+ def write(row: InternalRow): Unit = dataType match {
+ case st: StructType =>
+ writeObject(writeFields(row, st, rootFieldWriters))
+ case _ => throw new UnsupportedOperationException(
+ s"`JacksonGenerator` can only be used to write out a row when initialized with `StructType`.")
+ }
/**
- * Transforms multiple `InternalRow`s to JSON array using Jackson
+ * Transforms multiple `InternalRow`s or `MapData`s to JSON array using Jackson
*
- * @param array The array of rows to convert
+ * @param array The array of rows or maps to convert
*/
- def write(array: ArrayData): Unit = writeArray(writeArrayData(array, arrElementWriter))
+ def write(array: ArrayData): Unit = dataType match {
+ case st: StructType =>
+ try {
+ if (array.numElements() > 0) {
+ array.getStruct(0, st.length)
+ }
+ } catch {
+ case cce: ClassCastException =>
+ throw new UnsupportedOperationException(
+ s"`JacksonGenerator` can only be used to write out an array of struct " +
+ s"when initialized with `StructType`")
+ }
+ writeArray(writeArrayData(array, arrElementWriter))
+ case _: MapType =>
+ try {
--- End diff --
ditto.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137976262
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
@@ -677,14 +696,27 @@ case class StructsToJson(
override def checkInputDataTypes(): TypeCheckResult = child.dataType match {
case _: StructType | ArrayType(_: StructType, _) =>
try {
- JacksonUtils.verifySchema(rowSchema)
+ JacksonUtils.verifySchema(rowSchema.asInstanceOf[StructType])
+ TypeCheckResult.TypeCheckSuccess
+ } catch {
+ case e: UnsupportedOperationException =>
+ TypeCheckResult.TypeCheckFailure(e.getMessage)
+ }
+ case ArrayType(_: MapType, _) =>
+ TypeCheckResult.TypeCheckSuccess
+ case MapType(_: DataType, st: StructType, _: Boolean) =>
+ try {
+ JacksonUtils.verifySchema(st)
TypeCheckResult.TypeCheckSuccess
} catch {
case e: UnsupportedOperationException =>
TypeCheckResult.TypeCheckFailure(e.getMessage)
}
+ case _: MapType =>
--- End diff --
We need to revise `JacksonUtils` to check map type too.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81589 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81589/testReport)** for PR 18875 at commit [`8aa4307`](https://github.com/apache/spark/commit/8aa430749357f9e393b9b5856dbee7425af77302).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r132613913
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
@@ -659,13 +660,19 @@ case class StructsToJson(
(arr: Any) =>
gen.write(arr.asInstanceOf[ArrayData])
getAndReset()
+ case MapType(_: DataType, _: StructType, _: Boolean) =>
+ (map: Any) =>
+ val mapType = child.dataType.asInstanceOf[MapType]
+ gen.write(map.asInstanceOf[MapData], mapType)
+ getAndReset()
}
}
override def dataType: DataType = StringType
override def checkInputDataTypes(): TypeCheckResult = child.dataType match {
- case _: StructType | ArrayType(_: StructType, _) =>
+ case _: StructType | ArrayType(_: StructType, _) |
+ MapType(_: DataType, _: StructType, _: Boolean) =>
--- End diff --
I was thinking that the function name for `StructsToJson` might be similar word, but looks it is just `to_json`, there is no wording of struct. Yeah, if so, `StructsToJson` is not public API so I think it is fine to change it.
Because this is his first PR, I'd prefer less complexity for him. And to support an arbitrary Map type needs to change few more places and makes review process longer. It should be great if we can let this in and @goldmedal can work on another PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r138501335
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -26,20 +26,50 @@ import org.apache.spark.sql.catalyst.expressions.SpecializedGetters
import org.apache.spark.sql.catalyst.util.{ArrayData, DateTimeUtils, MapData}
import org.apache.spark.sql.types._
+/**
+ * `JackGenerator` can only be initialized with a `StructType` or a `MapType`.
+ * Once it is initialized with `StructType`, it can be used to write out a struct or an array of
+ * struct. Once it is initialized with `MapType`, it can be used to write out a map or an array
+ * of map. An exception will be thrown if trying to write out a struct if it is initialized with
+ * a `MapType`, and vice verse.
+ */
private[sql] class JacksonGenerator(
- schema: StructType,
+ dataType: DataType,
writer: Writer,
options: JSONOptions) {
// A `ValueWriter` is responsible for writing a field of an `InternalRow` to appropriate
// JSON data. Here we are using `SpecializedGetters` rather than `InternalRow` so that
// we can directly access data in `ArrayData` without the help of `SpecificMutableRow`.
private type ValueWriter = (SpecializedGetters, Int) => Unit
+ // `JackGenerator` can only be initialized with a `StructType` or a `MapType`.
+ require(dataType.isInstanceOf[StructType] | dataType.isInstanceOf[MapType],
--- End diff --
oh. Yes, you're right. This is my mistake. :(
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r131903872
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
@@ -638,6 +638,7 @@ case class StructsToJson(
lazy val rowSchema = child.dataType match {
--- End diff --
Could we add an examples in `ExpressionDescription` above at 611L like https://github.com/goldmedal/spark/commit/0cdcf9114527a2c359c25e46fd6556b3855bfb28#diff-6626026091295ad8c0dfb66ecbcd04b1R605?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r136995614
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -27,21 +27,45 @@ import org.apache.spark.sql.catalyst.util.{ArrayData, DateTimeUtils, MapData}
import org.apache.spark.sql.types._
private[sql] class JacksonGenerator(
- schema: StructType,
+ childType: DataType,
--- End diff --
This parameter looks weird to me. We don't want to tightly link `JacksonGenerator` to what this PR did.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137282128
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -22,24 +22,49 @@ import java.io.Writer
import com.fasterxml.jackson.core._
import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
import org.apache.spark.sql.catalyst.expressions.SpecializedGetters
import org.apache.spark.sql.catalyst.util.{ArrayData, DateTimeUtils, MapData}
import org.apache.spark.sql.types._
private[sql] class JacksonGenerator(
- schema: StructType,
+ dataType: DataType,
writer: Writer,
options: JSONOptions) {
// A `ValueWriter` is responsible for writing a field of an `InternalRow` to appropriate
// JSON data. Here we are using `SpecializedGetters` rather than `InternalRow` so that
// we can directly access data in `ArrayData` without the help of `SpecificMutableRow`.
private type ValueWriter = (SpecializedGetters, Int) => Unit
+ // `JackGenerator` only supports to write out a struct, an array of struct or an arbitrary map
--- End diff --
ok. I'll change it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81604 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81604/testReport)** for PR 18875 at commit [`67c3f45`](https://github.com/apache/spark/commit/67c3f4543c14180d1fd703b817073b9db3e24c98).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81263 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81263/testReport)** for PR 18875 at commit [`960fa3f`](https://github.com/apache/spark/commit/960fa3f58fd349b05e0be7c911488ea613fc2427).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r138228117
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -193,14 +223,35 @@ private[sql] class JacksonGenerator(
*
* @param row The row to convert
*/
- def write(row: InternalRow): Unit = writeObject(writeFields(row, schema, rootFieldWriters))
+ def write(row: InternalRow): Unit = {
+ writeObject(writeFields(
+ fieldWriters = rootFieldWriters,
+ row = row,
+ schema = dataType.asInstanceOf[StructType]))
+ }
+
+
+ /**
+ * Transforms multiple `InternalRow`s or `MapData`s to JSON array using Jackson
+ *
+ * @param array The array of rows or maps to convert
+ */
+ def write(array: ArrayData): Unit = writeArray(writeArrayData(
+ fieldWriter = arrElementWriter,
+ array = array
+ ))
--- End diff --
Let's change this one back to:
```scala
def write(array: ArrayData): Unit = writeArray(writeArrayData(array, arrElementWriter))
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r138500447
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -26,20 +26,50 @@ import org.apache.spark.sql.catalyst.expressions.SpecializedGetters
import org.apache.spark.sql.catalyst.util.{ArrayData, DateTimeUtils, MapData}
import org.apache.spark.sql.types._
+/**
+ * `JackGenerator` can only be initialized with a `StructType` or a `MapType`.
+ * Once it is initialized with `StructType`, it can be used to write out a struct or an array of
+ * struct. Once it is initialized with `MapType`, it can be used to write out a map or an array
+ * of map. An exception will be thrown if trying to write out a struct if it is initialized with
+ * a `MapType`, and vice verse.
+ */
private[sql] class JacksonGenerator(
- schema: StructType,
+ dataType: DataType,
writer: Writer,
options: JSONOptions) {
// A `ValueWriter` is responsible for writing a field of an `InternalRow` to appropriate
// JSON data. Here we are using `SpecializedGetters` rather than `InternalRow` so that
// we can directly access data in `ArrayData` without the help of `SpecificMutableRow`.
private type ValueWriter = (SpecializedGetters, Int) => Unit
+ // `JackGenerator` can only be initialized with a `StructType` or a `MapType`.
+ require(dataType.isInstanceOf[StructType] | dataType.isInstanceOf[MapType],
--- End diff --
Hm.. I think we should use `||` for short circuiting next time.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81611 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81611/testReport)** for PR 18875 at commit [`04753a4`](https://github.com/apache/spark/commit/04753a4683c306be5f48b9d9e85ad296e72a9161).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81380 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81380/testReport)** for PR 18875 at commit [`d466524`](https://github.com/apache/spark/commit/d466524e918361891ef406e4fe9d9b3b638054c3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80586/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137268446
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -127,7 +153,7 @@ private[sql] class JacksonGenerator(
(row: SpecializedGetters, ordinal: Int) =>
val v = row.get(ordinal, dataType)
sys.error(s"Failed to convert value $v (class of ${v.getClass}}) " +
- s"with the type of $dataType to JSON.")
+ s"with the type of $dataType to JSON.")
--- End diff --
Why change this?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #80596 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80596/testReport)** for PR 18875 at commit [`c64d9c4`](https://github.com/apache/spark/commit/c64d9c49c5f42b7a72b1570f00daa34a2843be1d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137284596
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -22,24 +22,49 @@ import java.io.Writer
import com.fasterxml.jackson.core._
import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
--- End diff --
Don't forget to remove this useless import.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/18875
retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81588 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81588/testReport)** for PR 18875 at commit [`b9a8c47`](https://github.com/apache/spark/commit/b9a8c4759fb6189005c02f30bb4108c28b941335).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137282534
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -193,14 +218,34 @@ private[sql] class JacksonGenerator(
*
* @param row The row to convert
*/
- def write(row: InternalRow): Unit = writeObject(writeFields(row, schema, rootFieldWriters))
+ def write(row: InternalRow): Unit = dataType match {
+ case st: StructType =>
+ writeObject(writeFields(row, st, rootFieldWriters))
+ case _ => throw new UnsupportedOperationException(
+ s"this api is only used when `JacksonGenerator` is initialized with `StructType`")
+ }
/**
* Transforms multiple `InternalRow`s to JSON array using Jackson
*
* @param array The array of rows to convert
*/
- def write(array: ArrayData): Unit = writeArray(writeArrayData(array, arrElementWriter))
+ def write(array: ArrayData): Unit = dataType match {
+ case _: StructType => writeArray(writeArrayData(array, arrElementWriter))
+ case _ => throw new UnsupportedOperationException(
+ s"this api is only used when `JacksonGenerator` is initialized with `StructType`")
--- End diff --
`JacksonGenerator` can only be used to write out an array when initialized with `StructType`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81632 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81632/testReport)** for PR 18875 at commit [`8835aca`](https://github.com/apache/spark/commit/8835aca5a5ccf420010b24e43189bfa8f1e8dc2a).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137284389
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
@@ -669,14 +679,25 @@ case class StructsToJson(
override def checkInputDataTypes(): TypeCheckResult = child.dataType match {
case _: StructType | ArrayType(_: StructType, _) =>
try {
- JacksonUtils.verifySchema(rowSchema)
+ JacksonUtils.verifySchema(rowSchema.asInstanceOf[StructType])
+ TypeCheckResult.TypeCheckSuccess
+ } catch {
+ case e: UnsupportedOperationException =>
+ TypeCheckResult.TypeCheckFailure(e.getMessage)
+ }
+ case MapType(_: DataType, st: StructType, _: Boolean) =>
+ try {
+ JacksonUtils.verifySchema(st)
TypeCheckResult.TypeCheckSuccess
} catch {
case e: UnsupportedOperationException =>
TypeCheckResult.TypeCheckFailure(e.getMessage)
}
+ case _: MapType =>
+ TypeCheckResult.TypeCheckSuccess
--- End diff --
This part of change looks verbose as `MapType` shows in two pattern cases. I will look into this part tomorrow.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81630/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81684/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137987296
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -193,14 +228,32 @@ private[sql] class JacksonGenerator(
*
* @param row The row to convert
*/
- def write(row: InternalRow): Unit = writeObject(writeFields(row, schema, rootFieldWriters))
+ def write(row: InternalRow): Unit = dataType match {
+ case st: StructType =>
+ writeObject(writeFields(row, st, rootFieldWriters))
+ case _ => throw new UnsupportedOperationException(
--- End diff --
oh I got you wrong. I thought you mean the matching in `rootFieldWeriters`. So we need to keep both of them?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r138232491
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -193,14 +223,35 @@ private[sql] class JacksonGenerator(
*
* @param row The row to convert
*/
- def write(row: InternalRow): Unit = writeObject(writeFields(row, schema, rootFieldWriters))
+ def write(row: InternalRow): Unit = {
+ writeObject(writeFields(
+ fieldWriters = rootFieldWriters,
+ row = row,
+ schema = dataType.asInstanceOf[StructType]))
+ }
+
+
+ /**
+ * Transforms multiple `InternalRow`s or `MapData`s to JSON array using Jackson
+ *
+ * @param array The array of rows or maps to convert
+ */
+ def write(array: ArrayData): Unit = writeArray(writeArrayData(
+ fieldWriter = arrElementWriter,
+ array = array
+ ))
/**
- * Transforms multiple `InternalRow`s to JSON array using Jackson
+ * Transforms a single `MapData` to JSON object using Jackson
*
- * @param array The array of rows to convert
+ * @param map a map to convert
*/
- def write(array: ArrayData): Unit = writeArray(writeArrayData(array, arrElementWriter))
+ def write(map: MapData): Unit = {
+ writeObject(writeMapData(
+ fieldWriter = mapElementWriter,
--- End diff --
And also add a similar comment in `write(row: InternalRow)`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r132614326
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala ---
@@ -186,6 +186,18 @@ class JsonFunctionsSuite extends QueryTest with SharedSQLContext {
Row("""[{"_1":1}]""") :: Nil)
}
+ test("to_json - map") {
+ val df1 = Seq(Map("a" -> Tuple1(1))).toDF("a")
+ val df2 = Seq(Map(Tuple1(1) -> Tuple1(1))).toDF("a")
+
+ checkAnswer(
+ df1.select(to_json($"a")),
+ Row("""{"a":{"_1":1}}""") :: Nil)
+ checkAnswer(
+ df2.select(to_json($"a")),
+ Row("""{"[0,1]":{"_1":1}}""") :: Nil)
--- End diff --
`[0,1]` is for the key `Tuple1(1)` ? I think it should be `[1]`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r132614025
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3106,9 +3106,9 @@ object functions {
}
/**
- * (Java-specific) Converts a column containing a `StructType` or `ArrayType` of `StructType`s
- * into a JSON string with the specified schema. Throws an exception, in the case of an
- * unsupported type.
+ * (Java-specific) Converts a column containing a `StructType`, `ArrayType` of `StructType`s
+ * or a `MapType` with a `StructType` Value into a JSON string with the specified schema.
--- End diff --
ditto.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81566 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81566/testReport)** for PR 18875 at commit [`7942039`](https://github.com/apache/spark/commit/7942039c28e7ef9f751b2f2a3ad0ef326f6abe7c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81616/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #80532 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80532/testReport)** for PR 18875 at commit [`f83a110`](https://github.com/apache/spark/commit/f83a11061b9905ffef8dbc8d58dfe7e8b0edaa86).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #80500 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80500/testReport)** for PR 18875 at commit [`43ec2dd`](https://github.com/apache/spark/commit/43ec2dd0704d4ef7cd013ffc0185de570072d172).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81182 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81182/testReport)** for PR 18875 at commit [`771e77a`](https://github.com/apache/spark/commit/771e77ab3ea23f672a668d4e171f8d1d1e956cc9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on the issue:
https://github.com/apache/spark/pull/18875
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r132875524
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
@@ -640,6 +644,7 @@ case class StructsToJson(
lazy val rowSchema = child.dataType match {
case st: StructType => st
case ArrayType(st: StructType, _) => st
+ case MapType(kt: DataType, st: StructType, _) => st
--- End diff --
little nit: `kt` -> `_`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #80850 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80850/testReport)** for PR 18875 at commit [`f49adcf`](https://github.com/apache/spark/commit/f49adcf1655f67c937dcb1cbe3dfc0763eb5e023).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r132875025
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
@@ -676,7 +687,8 @@ case class StructsToJson(
TypeCheckResult.TypeCheckFailure(e.getMessage)
}
case _ => TypeCheckResult.TypeCheckFailure(
- s"Input type ${child.dataType.simpleString} must be a struct or array of structs.")
+ s"Input type ${child.dataType.simpleString} must be a struct, array of structs or " +
+ s"map with a structs value.")
--- End diff --
little nit: `s` looks not required.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #80586 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80586/testReport)** for PR 18875 at commit [`d2a6d5e`](https://github.com/apache/spark/commit/d2a6d5e520cc6de36624b2f6dc7b730ee3b2eb8b).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137268116
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -22,24 +22,50 @@ import java.io.Writer
import com.fasterxml.jackson.core._
import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
import org.apache.spark.sql.catalyst.expressions.SpecializedGetters
import org.apache.spark.sql.catalyst.util.{ArrayData, DateTimeUtils, MapData}
import org.apache.spark.sql.types._
private[sql] class JacksonGenerator(
- schema: StructType,
+ dataType: DataType,
writer: Writer,
options: JSONOptions) {
// A `ValueWriter` is responsible for writing a field of an `InternalRow` to appropriate
// JSON data. Here we are using `SpecializedGetters` rather than `InternalRow` so that
// we can directly access data in `ArrayData` without the help of `SpecificMutableRow`.
private type ValueWriter = (SpecializedGetters, Int) => Unit
+ // `JackGenerator` only supports to write out a struct, an array of struct or an arbitrary map
+ dataType match {
+ case _: StructType | _: MapType =>
+ TypeCheckResult.TypeCheckSuccess
--- End diff --
We don't need `TypeCheckResult.TypeCheckSuccess`. Just no-op.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137977638
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -193,14 +228,32 @@ private[sql] class JacksonGenerator(
*
* @param row The row to convert
*/
- def write(row: InternalRow): Unit = writeObject(writeFields(row, schema, rootFieldWriters))
+ def write(row: InternalRow): Unit = dataType match {
+ case st: StructType =>
+ writeObject(writeFields(row, st, rootFieldWriters))
+ case _ => throw new UnsupportedOperationException(
--- End diff --
`ClassCastExceptoin` may not informative as `UnsupportedOperationException` here. I'd prefer to see `UnsupportedOperationException`. Let's keep this pattern matching for now.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137269586
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -127,7 +153,7 @@ private[sql] class JacksonGenerator(
(row: SpecializedGetters, ordinal: Int) =>
val v = row.get(ordinal, dataType)
sys.error(s"Failed to convert value $v (class of ${v.getClass}}) " +
- s"with the type of $dataType to JSON.")
+ s"with the type of $dataType to JSON.")
--- End diff --
ummm.. I think it maybe modified by IDE automatically. I'll recover it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81416 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81416/testReport)** for PR 18875 at commit [`24a7a72`](https://github.com/apache/spark/commit/24a7a726ae613a72ee746db56e53ccc48b8038b5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81380 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81380/testReport)** for PR 18875 at commit [`d466524`](https://github.com/apache/spark/commit/d466524e918361891ef406e4fe9d9b3b638054c3).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137227592
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -22,24 +22,44 @@ import java.io.Writer
import com.fasterxml.jackson.core._
import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
import org.apache.spark.sql.catalyst.expressions.SpecializedGetters
import org.apache.spark.sql.catalyst.util.{ArrayData, DateTimeUtils, MapData}
import org.apache.spark.sql.types._
private[sql] class JacksonGenerator(
- schema: StructType,
+ rowSchema: DataType,
writer: Writer,
options: JSONOptions) {
+
// A `ValueWriter` is responsible for writing a field of an `InternalRow` to appropriate
// JSON data. Here we are using `SpecializedGetters` rather than `InternalRow` so that
// we can directly access data in `ArrayData` without the help of `SpecificMutableRow`.
private type ValueWriter = (SpecializedGetters, Int) => Unit
+ // `JackGenerator` only supports to write out a struct, an array of struct or an arbitrary map
+ rowSchema match {
+ case _: StructType | _: MapType =>
+ TypeCheckResult.TypeCheckSuccess
+ case _ => TypeCheckResult.TypeCheckFailure(
+ s"Input type ${rowSchema.simpleString} must be a struct or a map")
+ }
+
// `ValueWriter`s for all fields of the schema
- private val rootFieldWriters: Array[ValueWriter] = schema.map(_.dataType).map(makeWriter).toArray
+ private lazy val rootFieldWriters: Array[ValueWriter] = {
+ rowSchema.asInstanceOf[StructType].map(_.dataType).map(makeWriter).toArray
+ }
+
// `ValueWriter` for array data storing rows of the schema.
- private val arrElementWriter: ValueWriter = (arr: SpecializedGetters, i: Int) => {
- writeObject(writeFields(arr.getStruct(i, schema.length), schema, rootFieldWriters))
+ private lazy val arrElementWriter: ValueWriter = {
--- End diff --
ditto.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137227920
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -193,14 +213,27 @@ private[sql] class JacksonGenerator(
*
* @param row The row to convert
*/
- def write(row: InternalRow): Unit = writeObject(writeFields(row, schema, rootFieldWriters))
+ def write(row: InternalRow): Unit = {
+ writeObject(writeFields(row, rowSchema.asInstanceOf[StructType], rootFieldWriters))
+ }
/**
* Transforms multiple `InternalRow`s to JSON array using Jackson
*
* @param array The array of rows to convert
*/
- def write(array: ArrayData): Unit = writeArray(writeArrayData(array, arrElementWriter))
+ def write(array: ArrayData): Unit = {
+ writeArray(writeArrayData(array, arrElementWriter))
--- End diff --
ditto.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81263/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81631 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81631/testReport)** for PR 18875 at commit [`be62e51`](https://github.com/apache/spark/commit/be62e51109c8a438fed8d787fb2986aeb1974f82).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81604 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81604/testReport)** for PR 18875 at commit [`67c3f45`](https://github.com/apache/spark/commit/67c3f4543c14180d1fd703b817073b9db3e24c98).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137976953
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -193,14 +228,32 @@ private[sql] class JacksonGenerator(
*
* @param row The row to convert
*/
- def write(row: InternalRow): Unit = writeObject(writeFields(row, schema, rootFieldWriters))
+ def write(row: InternalRow): Unit = dataType match {
+ case st: StructType =>
+ writeObject(writeFields(row, st, rootFieldWriters))
+ case _ => throw new UnsupportedOperationException(
--- End diff --
@viirya I found if we don't check it in here, it will throw `ClassCastExceptoin` from `writeObject()` firstly. So, maybe we don't need to check type in `rootFieldWriters` ?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81555 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81555/testReport)** for PR 18875 at commit [`723cd20`](https://github.com/apache/spark/commit/723cd20fbd4ad4ef6be513546f17f4629dcf3238).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137986264
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
@@ -677,14 +696,25 @@ case class StructsToJson(
override def checkInputDataTypes(): TypeCheckResult = child.dataType match {
case _: StructType | ArrayType(_: StructType, _) =>
try {
- JacksonUtils.verifySchema(rowSchema)
+ JacksonUtils.verifySchema(rowSchema.asInstanceOf[StructType])
+ TypeCheckResult.TypeCheckSuccess
+ } catch {
+ case e: UnsupportedOperationException =>
+ TypeCheckResult.TypeCheckFailure(e.getMessage)
+ }
+ case _: MapType | ArrayType(_: MapType, _) =>
+ // TODO: let `JacksonUtils.verifySchema` verify a `MapType`
+ try {
+ val st = StructType(StructField("a", rowSchema.asInstanceOf[MapType]) :: Nil)
+ JacksonUtils.verifySchema(st)
TypeCheckResult.TypeCheckSuccess
} catch {
case e: UnsupportedOperationException =>
TypeCheckResult.TypeCheckFailure(e.getMessage)
}
case _ => TypeCheckResult.TypeCheckFailure(
- s"Input type ${child.dataType.simpleString} must be a struct or array of structs.")
+ s"Input type ${child.dataType.simpleString} must be a struct, array of structs or " +
+ s"an arbitrary map.")
--- End diff --
`an arbitrary map` -> `a map or array of map.`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81616 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81616/testReport)** for PR 18875 at commit [`b8bf38c`](https://github.com/apache/spark/commit/b8bf38cda5f3b03473f5bd1f1c4c8599e342b869).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137227950
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -193,14 +213,27 @@ private[sql] class JacksonGenerator(
*
* @param row The row to convert
*/
- def write(row: InternalRow): Unit = writeObject(writeFields(row, schema, rootFieldWriters))
+ def write(row: InternalRow): Unit = {
+ writeObject(writeFields(row, rowSchema.asInstanceOf[StructType], rootFieldWriters))
+ }
/**
* Transforms multiple `InternalRow`s to JSON array using Jackson
*
* @param array The array of rows to convert
*/
- def write(array: ArrayData): Unit = writeArray(writeArrayData(array, arrElementWriter))
+ def write(array: ArrayData): Unit = {
+ writeArray(writeArrayData(array, arrElementWriter))
+ }
+
+ /**
+ * Transforms a `MapData` to JSON object using Jackson
+ *
+ * @param map a map to convert
+ */
+ def write(map: MapData): Unit = {
+ writeObject(writeMapData(map, rowSchema.asInstanceOf[MapType], mapElementWriter))
--- End diff --
ditto.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r138057979
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
@@ -677,14 +696,25 @@ case class StructsToJson(
override def checkInputDataTypes(): TypeCheckResult = child.dataType match {
case _: StructType | ArrayType(_: StructType, _) =>
try {
- JacksonUtils.verifySchema(rowSchema)
+ JacksonUtils.verifySchema(rowSchema.asInstanceOf[StructType])
+ TypeCheckResult.TypeCheckSuccess
+ } catch {
+ case e: UnsupportedOperationException =>
+ TypeCheckResult.TypeCheckFailure(e.getMessage)
+ }
+ case _: MapType | ArrayType(_: MapType, _) =>
+ // TODO: let `JacksonUtils.verifySchema` verify a `MapType`
+ try {
+ val st = StructType(StructField("a", rowSchema.asInstanceOf[MapType]) :: Nil)
+ JacksonUtils.verifySchema(st)
TypeCheckResult.TypeCheckSuccess
} catch {
case e: UnsupportedOperationException =>
TypeCheckResult.TypeCheckFailure(e.getMessage)
}
case _ => TypeCheckResult.TypeCheckFailure(
- s"Input type ${child.dataType.simpleString} must be a struct or array of structs.")
+ s"Input type ${child.dataType.simpleString} must be a struct, array of structs or " +
+ s"a map or array of map.")
--- End diff --
Looks `s` could be removed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/18875
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r132609949
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
@@ -659,13 +660,19 @@ case class StructsToJson(
(arr: Any) =>
gen.write(arr.asInstanceOf[ArrayData])
getAndReset()
+ case MapType(_: DataType, _: StructType, _: Boolean) =>
+ (map: Any) =>
+ val mapType = child.dataType.asInstanceOf[MapType]
+ gen.write(map.asInstanceOf[MapData], mapType)
+ getAndReset()
}
}
override def dataType: DataType = StringType
override def checkInputDataTypes(): TypeCheckResult = child.dataType match {
- case _: StructType | ArrayType(_: StructType, _) =>
+ case _: StructType | ArrayType(_: StructType, _) |
+ MapType(_: DataType, _: StructType, _: Boolean) =>
--- End diff --
Is there no compatibility issue if we rename the expression?
I've looked around and I think it should not be difficult to support `MapType` like `StructType`, although we needs to change few more places.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81625 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81625/testReport)** for PR 18875 at commit [`b71b56a`](https://github.com/apache/spark/commit/b71b56ab49ba048f397459a18f946422d9440b29).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137975624
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/json/JacksonGeneratorSuite.scala ---
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.json
+
+import java.io.CharArrayWriter
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, DateTimeUtils, GenericArrayData}
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+class JacksonGeneratorSuite extends SparkFunSuite {
+
+ val gmtId = DateTimeUtils.TimeZoneGMT.getID
+ val option = new JSONOptions(Map.empty, gmtId)
+ val writer = new CharArrayWriter()
+ def getAndReset(gen: JacksonGenerator): UTF8String = {
+ gen.flush()
+ val json = writer.toString
+ writer.reset()
--- End diff --
I don't prefer tests depending each others. I think we can just have a writer for each test, instead of reusing it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81620 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81620/testReport)** for PR 18875 at commit [`b8ad442`](https://github.com/apache/spark/commit/b8ad442437390611c63d8b0705d02c3ca3ba3c9c).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18875
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81589/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81382 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81382/testReport)** for PR 18875 at commit [`3ebbe67`](https://github.com/apache/spark/commit/3ebbe67e059dfb6a004ff50f3c661f6319d616b8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18875
**[Test build #81542 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81542/testReport)** for PR 18875 at commit [`1df28ec`](https://github.com/apache/spark/commit/1df28ec200fd46a001b0fea9597f8b9659ea94f4).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r138500520
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
@@ -617,6 +618,14 @@ case class JsonToStructs(
{"time":"26/08/2015"}
> SELECT _FUNC_(array(named_struct('a', 1, 'b', 2));
[{"a":1,"b":2}]
+ > SELECT _FUNC_(map('a',named_struct('b',1)));
--- End diff --
little nit: `... ',n ...` -> `... ', n ...`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137224911
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala ---
@@ -640,6 +644,7 @@ case class StructsToJson(
lazy val rowSchema = child.dataType match {
case st: StructType => st
case ArrayType(st: StructType, _) => st
+ case MapType(_: DataType, _: DataType, _) => child.dataType
--- End diff --
case mt: MapType => mt
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137978728
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -193,14 +228,32 @@ private[sql] class JacksonGenerator(
*
* @param row The row to convert
*/
- def write(row: InternalRow): Unit = writeObject(writeFields(row, schema, rootFieldWriters))
+ def write(row: InternalRow): Unit = dataType match {
+ case st: StructType =>
+ writeObject(writeFields(row, st, rootFieldWriters))
+ case _ => throw new UnsupportedOperationException(
--- End diff --
ok got it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137280095
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -22,24 +22,49 @@ import java.io.Writer
import com.fasterxml.jackson.core._
import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
import org.apache.spark.sql.catalyst.expressions.SpecializedGetters
import org.apache.spark.sql.catalyst.util.{ArrayData, DateTimeUtils, MapData}
import org.apache.spark.sql.types._
private[sql] class JacksonGenerator(
- schema: StructType,
+ dataType: DataType,
writer: Writer,
options: JSONOptions) {
// A `ValueWriter` is responsible for writing a field of an `InternalRow` to appropriate
// JSON data. Here we are using `SpecializedGetters` rather than `InternalRow` so that
// we can directly access data in `ArrayData` without the help of `SpecificMutableRow`.
private type ValueWriter = (SpecializedGetters, Int) => Unit
+ // `JackGenerator` only supports to write out a struct, an array of struct or an arbitrary map
--- End diff --
`JackGenerator` can only be initialized with a StructType or a MapType.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18875: [SPARK-21513][SQL] Allow UDF to_json support converting ...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on the issue:
https://github.com/apache/spark/pull/18875
@HyukjinKwon ok. I got it. Thanks =)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137287045
--- Diff: sql/core/src/test/resources/sql-tests/results/cross-join.sql.out ---
@@ -128,6 +128,7 @@ two 2 two 2 one 1 two 2
two 2 two 2 three 3 two 2
two 2 two 2 two 2 two 2
+
--- End diff --
I don't saw you commit change to `cross-join.sql`, why we have change to `cross-join.sql.out`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by goldmedal <gi...@git.apache.org>.
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137959771
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala ---
@@ -193,14 +228,58 @@ private[sql] class JacksonGenerator(
*
* @param row The row to convert
*/
- def write(row: InternalRow): Unit = writeObject(writeFields(row, schema, rootFieldWriters))
+ def write(row: InternalRow): Unit = dataType match {
+ case st: StructType =>
+ writeObject(writeFields(row, st, rootFieldWriters))
+ case _ => throw new UnsupportedOperationException(
+ s"`JacksonGenerator` can only be used to write out a row when initialized with `StructType`.")
+ }
/**
- * Transforms multiple `InternalRow`s to JSON array using Jackson
+ * Transforms multiple `InternalRow`s or `MapData`s to JSON array using Jackson
*
- * @param array The array of rows to convert
+ * @param array The array of rows or maps to convert
*/
- def write(array: ArrayData): Unit = writeArray(writeArrayData(array, arrElementWriter))
+ def write(array: ArrayData): Unit = dataType match {
+ case st: StructType =>
+ try {
--- End diff --
I think you're right. If the caller has an error calling, it will also throw a `ClassCastException` to remind the caller. So we're unnecessary to check it.
I'll remove it. Thanks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r138102302
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala ---
@@ -180,10 +180,30 @@ class JsonFunctionsSuite extends QueryTest with SharedSQLContext {
test("to_json - array") {
val df = Seq(Tuple1(Tuple1(1) :: Nil)).toDF("a")
+ val df2 = Seq(Tuple1(Map("a" -> 1) :: Nil)).toDF("a")
checkAnswer(
df.select(to_json($"a")),
Row("""[{"_1":1}]""") :: Nil)
+ checkAnswer(
+ df2.select(to_json($"a")),
+ Row("""[{"a":1}]""") :: Nil)
+ }
+
+ test("to_json - map") {
+ val df1 = Seq(Map("a" -> Tuple1(1))).toDF("a")
+ val df2 = Seq(Map(Tuple1(1) -> Tuple1(1))).toDF("a")
+ val df3 = Seq(Map("a" -> 1)).toDF("a")
+
+ checkAnswer(
+ df1.select(to_json($"a")),
+ Row("""{"a":{"_1":1}}""") :: Nil)
+ checkAnswer(
+ df2.select(to_json($"a")),
+ Row("""{"[0,1]":{"_1":1}}""") :: Nil)
--- End diff --
Yes, I was thinking that showing debugging (internal formatted) strings is rather a bug and actually we might have to fix if possible, as I think we have pretty `toString` format for public API `Row` class.
I remember we eventually decided to fix `TimestampType` and `DataType` in JSON, which were printed as longs and integers, to human readable formats, SPARK-16216. I think there was a discussion that they should be written as are (long and int) vs human readable format at that time if I understood correctly.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org