You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "chenhao-db (via GitHub)" <gi...@apache.org> on 2024/04/02 02:29:25 UTC
[PR] [SPARK-47680][SQL] Add variant_explode expression. [spark]
chenhao-db opened a new pull request, #45805:
URL: https://github.com/apache/spark/pull/45805
### What changes were proposed in this pull request?
This PR adds a new `VariantExplode` expression. It separates a variant object/array into multiple rows containing its fields/elements. Its result schema is `struct<pos int, key string, value variant>`. `pos` is the position of the field/element in its parent object/array, and `value` is the field/element value. `key` is the field name when exploding a variant object, or is NULL when exploding a variant array. It ignores any input that is not a variant array/object, including SQL NULL, variant null, and any other variant values.
It is exposed as two SQL expressions, `variant_explode` and `variant_explode_outer`. The only difference is that whenever `variant_explode` produces zero output row for an input row, `variant_explode_outer` will produce one output row containing `{NULL, NULL, NULL}`.
Usage examples:
```
> SELECT _FUNC_(parse_json('["hello", "world"]'));
0 NULL "hello"
1 NULL "world"
> SELECT _FUNC_(parse_json('{"a": true, "b": 3.14}'));
0 a true
1 b 3.14
```
### How was this patch tested?
Unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47680][SQL] Add variant_explode expression. [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #45805:
URL: https://github.com/apache/spark/pull/45805#discussion_r1555178869
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala:
##########
@@ -822,6 +822,8 @@ object FunctionRegistry {
expression[ParseJson]("parse_json"),
expressionBuilder("variant_get", VariantGetExpressionBuilder),
expressionBuilder("try_variant_get", TryVariantGetExpressionBuilder),
+ expression[VariantExplode]("variant_explode"),
+ expressionGeneratorOuter[VariantExplode]("variant_explode_outer"),
Review Comment:
It's not recommended to use the hive style generator SQL syntax anymore. Can we remove it and only register it as TVF?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47680][SQL] Add variant_explode expression. [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #45805:
URL: https://github.com/apache/spark/pull/45805#issuecomment-2047758862
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47680][SQL] Add variant_explode expression. [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #45805:
URL: https://github.com/apache/spark/pull/45805#discussion_r1555333257
##########
sql/core/src/test/scala/org/apache/spark/sql/VariantSuite.scala:
##########
@@ -298,4 +300,28 @@ class VariantSuite extends QueryTest with SharedSparkSession {
}
assert(ex.getErrorClass == "DATATYPE_MISMATCH.INVALID_ORDERING_TYPE")
}
+
+ test("variant_explode") {
+ def check(input: String, expected: Seq[Row]): Unit = {
+ withView("v") {
+ Seq(input).toDF("json").createOrReplaceTempView("v")
+ checkAnswer(sql("select pos, key, to_json(value) from v, " +
+ "lateral variant_explode(parse_json(json))"), expected)
+ val expectedOuter = if (expected.isEmpty) Seq(Row(null, null, null)) else expected
+ checkAnswer(sql("select pos, key, to_json(value) from v, " +
+ "lateral variant_explode_outer(parse_json(json))"), expectedOuter)
+ }
+ }
+
+ Seq("true", "false").foreach { codegenEnabled =>
+ withSQLConf(SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> codegenEnabled) {
+ check(null, Nil)
+ check("1", Nil)
Review Comment:
@srielau shall we fail if the input variant is not array or struct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47680][SQL] Add variant_explode expression. [spark]
Posted by "chenhao-db (via GitHub)" <gi...@apache.org>.
chenhao-db commented on code in PR #45805:
URL: https://github.com/apache/spark/pull/45805#discussion_r1558003489
##########
sql/core/src/test/scala/org/apache/spark/sql/VariantSuite.scala:
##########
@@ -298,4 +300,28 @@ class VariantSuite extends QueryTest with SharedSparkSession {
}
assert(ex.getErrorClass == "DATATYPE_MISMATCH.INVALID_ORDERING_TYPE")
}
+
+ test("variant_explode") {
+ def check(input: String, expected: Seq[Row]): Unit = {
+ withView("v") {
+ Seq(input).toDF("json").createOrReplaceTempView("v")
+ checkAnswer(sql("select pos, key, to_json(value) from v, " +
+ "lateral variant_explode(parse_json(json))"), expected)
+ val expectedOuter = if (expected.isEmpty) Seq(Row(null, null, null)) else expected
+ checkAnswer(sql("select pos, key, to_json(value) from v, " +
+ "lateral variant_explode_outer(parse_json(json))"), expectedOuter)
+ }
+ }
+
+ Seq("true", "false").foreach { codegenEnabled =>
+ withSQLConf(SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> codegenEnabled) {
+ check(null, Nil)
+ check("1", Nil)
Review Comment:
As a side note, Snowflake's [flatten](https://docs.snowflake.com/en/sql-reference/functions/flatten) returns an empty result set rather than throwing an error if the variant is not array or struct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47680][SQL] Add variant_explode expression. [spark]
Posted by "chenhao-db (via GitHub)" <gi...@apache.org>.
chenhao-db commented on code in PR #45805:
URL: https://github.com/apache/spark/pull/45805#discussion_r1555298791
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala:
##########
@@ -822,6 +822,8 @@ object FunctionRegistry {
expression[ParseJson]("parse_json"),
expressionBuilder("variant_get", VariantGetExpressionBuilder),
expressionBuilder("try_variant_get", TryVariantGetExpressionBuilder),
+ expression[VariantExplode]("variant_explode"),
+ expressionGeneratorOuter[VariantExplode]("variant_explode_outer"),
Review Comment:
Done.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala:
##########
@@ -403,3 +404,85 @@ object VariantGetExpressionBuilder extends VariantGetExpressionBuilderBase(true)
)
// scalastyle:on line.size.limit
object TryVariantGetExpressionBuilder extends VariantGetExpressionBuilderBase(false)
+
+// scalastyle:off line.size.limit line.contains.tab
+@ExpressionDescription(
+ usage = "_FUNC_(expr) - It separates a variant object/array into multiple rows containing its fields/elements. Its result schema is `struct<pos int, key string, value variant>`. `pos` is the position of the field/element in its parent object/array, and `value` is the field/element value. `key` is the field name when exploding a variant object, or is NULL when exploding a variant array. It ignores any input that is not a variant array/object, including SQL NULL, variant null, and any other variant values.",
+ examples = """
+ Examples:
+ > SELECT _FUNC_(parse_json('["hello", "world"]'));
+ 0 NULL "hello"
+ 1 NULL "world"
+ > SELECT _FUNC_(parse_json('{"a": true, "b": 3.14}'));
+ 0 a true
+ 1 b 3.14
+ """,
+ since = "4.0.0",
+ group = "variant_funcs")
+// scalastyle:on line.size.limit line.contains.tab
+case class VariantExplode(child: Expression) extends UnaryExpression with Generator
+ with ExpectsInputTypes {
+ override def inputTypes: Seq[AbstractDataType] = Seq(VariantType)
+
+ override def prettyName: String = "variant_explode"
+
+ override protected def withNewChildInternal(newChild: Expression): VariantExplode =
+ copy(child = newChild)
+
+ override def eval(input: InternalRow): IterableOnce[InternalRow] = {
+ val inputVariant = child.eval(input).asInstanceOf[VariantVal]
+ VariantExplode.variantExplode(inputVariant, inputVariant == null)
+ }
+
+ override protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val childCode = child.genCode(ctx)
+ val code = code"""
+ ${childCode.code}
+ scala.collection.Seq<InternalRow> ${ev.value} =
+ org.apache.spark.sql.catalyst.expressions.variant.VariantExplode.variantExplode(
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47680][SQL] Add variant_explode expression. [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #45805:
URL: https://github.com/apache/spark/pull/45805#discussion_r1557992983
##########
sql/core/src/test/scala/org/apache/spark/sql/VariantSuite.scala:
##########
@@ -298,4 +300,28 @@ class VariantSuite extends QueryTest with SharedSparkSession {
}
assert(ex.getErrorClass == "DATATYPE_MISMATCH.INVALID_ORDERING_TYPE")
}
+
+ test("variant_explode") {
+ def check(input: String, expected: Seq[Row]): Unit = {
+ withView("v") {
+ Seq(input).toDF("json").createOrReplaceTempView("v")
+ checkAnswer(sql("select pos, key, to_json(value) from v, " +
+ "lateral variant_explode(parse_json(json))"), expected)
+ val expectedOuter = if (expected.isEmpty) Seq(Row(null, null, null)) else expected
+ checkAnswer(sql("select pos, key, to_json(value) from v, " +
+ "lateral variant_explode_outer(parse_json(json))"), expectedOuter)
+ }
+ }
+
+ Seq("true", "false").foreach { codegenEnabled =>
+ withSQLConf(SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> codegenEnabled) {
+ check(null, Nil)
+ check("1", Nil)
Review Comment:
Yeah, I think that's safest. Otherwise we would have to return an empty result set?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47680][SQL] Add variant_explode expression. [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #45805:
URL: https://github.com/apache/spark/pull/45805#discussion_r1555179535
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala:
##########
@@ -403,3 +404,85 @@ object VariantGetExpressionBuilder extends VariantGetExpressionBuilderBase(true)
)
// scalastyle:on line.size.limit
object TryVariantGetExpressionBuilder extends VariantGetExpressionBuilderBase(false)
+
+// scalastyle:off line.size.limit line.contains.tab
+@ExpressionDescription(
+ usage = "_FUNC_(expr) - It separates a variant object/array into multiple rows containing its fields/elements. Its result schema is `struct<pos int, key string, value variant>`. `pos` is the position of the field/element in its parent object/array, and `value` is the field/element value. `key` is the field name when exploding a variant object, or is NULL when exploding a variant array. It ignores any input that is not a variant array/object, including SQL NULL, variant null, and any other variant values.",
+ examples = """
+ Examples:
+ > SELECT _FUNC_(parse_json('["hello", "world"]'));
Review Comment:
let's use the SQL standard TVF syntax to invoke it in the example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47680][SQL] Add variant_explode expression. [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #45805:
URL: https://github.com/apache/spark/pull/45805#discussion_r1555180608
##########
sql/core/src/test/scala/org/apache/spark/sql/VariantSuite.scala:
##########
@@ -298,4 +300,26 @@ class VariantSuite extends QueryTest with SharedSparkSession {
}
assert(ex.getErrorClass == "DATATYPE_MISMATCH.INVALID_ORDERING_TYPE")
}
+
+ test("variant_explode") {
+ def check(input: String, expected: Seq[Row]): Unit = {
+ val df = Seq(input).toDF("json")
+ checkAnswer(df.selectExpr("variant_explode(parse_json(json))")
Review Comment:
ditto, please use the SQL standard TVF syntax to invoke it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47680][SQL] Add variant_explode expression. [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #45805:
URL: https://github.com/apache/spark/pull/45805#discussion_r1556857735
##########
sql/core/src/test/scala/org/apache/spark/sql/VariantSuite.scala:
##########
@@ -298,4 +300,28 @@ class VariantSuite extends QueryTest with SharedSparkSession {
}
assert(ex.getErrorClass == "DATATYPE_MISMATCH.INVALID_ORDERING_TYPE")
}
+
+ test("variant_explode") {
+ def check(input: String, expected: Seq[Row]): Unit = {
+ withView("v") {
+ Seq(input).toDF("json").createOrReplaceTempView("v")
+ checkAnswer(sql("select pos, key, to_json(value) from v, " +
+ "lateral variant_explode(parse_json(json))"), expected)
+ val expectedOuter = if (expected.isEmpty) Seq(Row(null, null, null)) else expected
+ checkAnswer(sql("select pos, key, to_json(value) from v, " +
+ "lateral variant_explode_outer(parse_json(json))"), expectedOuter)
+ }
+ }
+
+ Seq("true", "false").foreach { codegenEnabled =>
+ withSQLConf(SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> codegenEnabled) {
+ check(null, Nil)
+ check("1", Nil)
Review Comment:
Why? An INT can cast to a VARIANT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47680][SQL] Add variant_explode expression. [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #45805: [SPARK-47680][SQL] Add variant_explode expression.
URL: https://github.com/apache/spark/pull/45805
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47680][SQL] Add variant_explode expression. [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #45805:
URL: https://github.com/apache/spark/pull/45805#discussion_r1556871731
##########
sql/core/src/test/scala/org/apache/spark/sql/VariantSuite.scala:
##########
@@ -298,4 +300,28 @@ class VariantSuite extends QueryTest with SharedSparkSession {
}
assert(ex.getErrorClass == "DATATYPE_MISMATCH.INVALID_ORDERING_TYPE")
}
+
+ test("variant_explode") {
+ def check(input: String, expected: Seq[Row]): Unit = {
+ withView("v") {
+ Seq(input).toDF("json").createOrReplaceTempView("v")
+ checkAnswer(sql("select pos, key, to_json(value) from v, " +
+ "lateral variant_explode(parse_json(json))"), expected)
+ val expectedOuter = if (expected.isEmpty) Seq(Row(null, null, null)) else expected
+ checkAnswer(sql("select pos, key, to_json(value) from v, " +
+ "lateral variant_explode_outer(parse_json(json))"), expectedOuter)
+ }
+ }
+
+ Seq("true", "false").foreach { codegenEnabled =>
+ withSQLConf(SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> codegenEnabled) {
+ check(null, Nil)
+ check("1", Nil)
Review Comment:
oh, this is not a cast, this is exploding the input variant "1". The working example is listed below: `{"a": [1, 2, 3], "b": true}`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47680][SQL] Add variant_explode expression. [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #45805:
URL: https://github.com/apache/spark/pull/45805#discussion_r1555180077
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala:
##########
@@ -403,3 +404,85 @@ object VariantGetExpressionBuilder extends VariantGetExpressionBuilderBase(true)
)
// scalastyle:on line.size.limit
object TryVariantGetExpressionBuilder extends VariantGetExpressionBuilderBase(false)
+
+// scalastyle:off line.size.limit line.contains.tab
+@ExpressionDescription(
+ usage = "_FUNC_(expr) - It separates a variant object/array into multiple rows containing its fields/elements. Its result schema is `struct<pos int, key string, value variant>`. `pos` is the position of the field/element in its parent object/array, and `value` is the field/element value. `key` is the field name when exploding a variant object, or is NULL when exploding a variant array. It ignores any input that is not a variant array/object, including SQL NULL, variant null, and any other variant values.",
+ examples = """
+ Examples:
+ > SELECT _FUNC_(parse_json('["hello", "world"]'));
+ 0 NULL "hello"
+ 1 NULL "world"
+ > SELECT _FUNC_(parse_json('{"a": true, "b": 3.14}'));
+ 0 a true
+ 1 b 3.14
+ """,
+ since = "4.0.0",
+ group = "variant_funcs")
+// scalastyle:on line.size.limit line.contains.tab
+case class VariantExplode(child: Expression) extends UnaryExpression with Generator
+ with ExpectsInputTypes {
+ override def inputTypes: Seq[AbstractDataType] = Seq(VariantType)
+
+ override def prettyName: String = "variant_explode"
+
+ override protected def withNewChildInternal(newChild: Expression): VariantExplode =
+ copy(child = newChild)
+
+ override def eval(input: InternalRow): IterableOnce[InternalRow] = {
+ val inputVariant = child.eval(input).asInstanceOf[VariantVal]
+ VariantExplode.variantExplode(inputVariant, inputVariant == null)
+ }
+
+ override protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val childCode = child.genCode(ctx)
+ val code = code"""
+ ${childCode.code}
+ scala.collection.Seq<InternalRow> ${ev.value} =
+ org.apache.spark.sql.catalyst.expressions.variant.VariantExplode.variantExplode(
Review Comment:
let's not hardcode the class name. use `classOf[VariantExplode].getName`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47680][SQL] Add variant_explode expression. [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #45805:
URL: https://github.com/apache/spark/pull/45805#discussion_r1559582943
##########
sql/core/src/test/scala/org/apache/spark/sql/VariantSuite.scala:
##########
@@ -298,4 +300,28 @@ class VariantSuite extends QueryTest with SharedSparkSession {
}
assert(ex.getErrorClass == "DATATYPE_MISMATCH.INVALID_ORDERING_TYPE")
}
+
+ test("variant_explode") {
+ def check(input: String, expected: Seq[Row]): Unit = {
+ withView("v") {
+ Seq(input).toDF("json").createOrReplaceTempView("v")
+ checkAnswer(sql("select pos, key, to_json(value) from v, " +
+ "lateral variant_explode(parse_json(json))"), expected)
+ val expectedOuter = if (expected.isEmpty) Seq(Row(null, null, null)) else expected
+ checkAnswer(sql("select pos, key, to_json(value) from v, " +
+ "lateral variant_explode_outer(parse_json(json))"), expectedOuter)
+ }
+ }
+
+ Seq("true", "false").foreach { codegenEnabled =>
+ withSQLConf(SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> codegenEnabled) {
+ check(null, Nil)
+ check("1", Nil)
Review Comment:
I can live with an empty result set, given the precedence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org