You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "chenhao-db (via GitHub)" <gi...@apache.org> on 2024/04/18 21:51:22 UTC
[PR] [SPARK-47890][CONNECT][PYTHON] Add variant functions to Scala and Python. [spark]
chenhao-db opened a new pull request, #46123:
URL: https://github.com/apache/spark/pull/46123
### What changes were proposed in this pull request?
Add the following functions:
- is_variant_null
- variant_get
- try_variant_get
- schema_of_variant
- schema_of_variant_agg
to:
- Scala API
- Python API
- Spark Connect Scala Client
- Spark Connect Python Client
### Why are the changes needed?
Improve the usability of variant functions.
### Does this PR introduce _any_ user-facing change?
Yes, new functions
### How was this patch tested?
Unit tests.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47890][CONNECT][PYTHON] Add variant functions to Scala and Python. [spark]
Posted by "chenhao-db (via GitHub)" <gi...@apache.org>.
chenhao-db commented on code in PR #46123:
URL: https://github.com/apache/spark/pull/46123#discussion_r1574033907
##########
python/pyspark/sql/tests/test_functions.py:
##########
@@ -1315,6 +1315,35 @@ def test_parse_json(self):
self.assertEqual("""{"a":1}""", actual["var"])
self.assertEqual("""{"b":[{"c":"str2"}]}""", actual["var_lit"])
+ def test_variant_expressions(self):
+ df = self.spark.createDataFrame([Row(json="""{ "a" : 1 }"""), Row(json="""{ "b" : 2 }""")])
+ v = F.parse_json(df.json)
+
+ def check(resultDf, expected):
+ self.assertEqual([r[0] for r in resultDf.collect()], expected)
+
+ check(df.select(F.is_variant_null(v)), [False, False])
+ check(df.select(F.schema_of_variant(v)), ["STRUCT<a: BIGINT>", "STRUCT<b: BIGINT>"])
+ check(df.select(F.schema_of_variant_agg(v)), ["STRUCT<a: BIGINT, b: BIGINT>"])
+
+ check(df.select(F.variant_get(v, "$.a", "int")), [1, None])
+ check(df.select(F.variant_get(v, "$.b", "int")), [None, 2])
+ check(df.select(F.variant_get(v, "$.a", "double")), [1.0, None])
+
+ with self.assertRaises(SparkRuntimeException) as ex:
+ df.select(F.variant_get(v, "$.a", "binary")).collect()
Review Comment:
> only `variant_get` cast an exception, right?
Correct. We don't need to include more test cases in this PR because they have already been extensively tested in [VariantExpressionSuite](https://github.com/apache/spark/blob/9f34b8eca2f3578367352ec28532e29fe3a91864/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionSuite.scala#L251).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47890][CONNECT][PYTHON] Add variant functions to Scala and Python. [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #46123:
URL: https://github.com/apache/spark/pull/46123#discussion_r1571680929
##########
sql/core/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -6604,6 +6604,71 @@ object functions {
*/
def parse_json(json: Column): Column = Column.fn("parse_json", json)
+ /**
+ * Check if a variant value is a variant null. Returns true if and only if the input is a
+ * variant null and false otherwise (including in the case of SQL NULL).
+ *
+ * @param v
+ * a variant column.
+ * @group variant_funcs
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47890][CONNECT][PYTHON] Add variant functions to Scala and Python. [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #46123:
URL: https://github.com/apache/spark/pull/46123#discussion_r1571680727
##########
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -6975,6 +6975,71 @@ object functions {
*/
def parse_json(json: Column): Column = Column.fn("parse_json", json)
+ /**
+ * Check if a variant value is a variant null. Returns true if and only if the input is a
+ * variant null and false otherwise (including in the case of SQL NULL).
+ *
+ * @param v
+ * a variant column.
+ * @group variant_funcs
Review Comment:
it seems we don't have this `variant_funcs` group
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47890][CONNECT][PYTHON] Add variant functions to Scala and Python. [spark]
Posted by "chenhao-db (via GitHub)" <gi...@apache.org>.
chenhao-db commented on PR #46123:
URL: https://github.com/apache/spark/pull/46123#issuecomment-2068317353
@zhengruifeng @LuciferYang Could you help take another look? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47890][CONNECT][PYTHON] Add variant functions to Scala and Python. [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #46123:
URL: https://github.com/apache/spark/pull/46123#discussion_r1571532818
##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -15452,6 +15452,135 @@ def parse_json(
return _invoke_function("parse_json", _to_java_column(col))
+@_try_remote_functions
+def is_variant_null(v: "ColumnOrName") -> Column:
+ """
+ Check if a variant value is a variant null. Returns true if and only if the input is a variant
+ null and false otherwise (including in the case of SQL NULL).
+
+ .. versionadded:: 4.0.0
+
Review Comment:
Can we add `Returns` section (https://numpydoc.readthedocs.io/en/latest/format.html#returns)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47890][CONNECT][PYTHON] Add variant functions to Scala and Python. [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #46123:
URL: https://github.com/apache/spark/pull/46123#discussion_r1571874614
##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/PlanGenerationTestSuite.scala:
##########
@@ -2485,6 +2485,30 @@ class PlanGenerationTestSuite
fn.to_json(fn.col("d"), Map(("timestampFormat", "dd/MM/yyyy")))
}
+ functionTest("parse_json") {
Review Comment:
The corresponding `.explain` file is missing, it needs to be generated using `ProtoToParsedPlanTestSuite`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47890][CONNECT][PYTHON] Add variant functions to Scala and Python. [spark]
Posted by "chenhao-db (via GitHub)" <gi...@apache.org>.
chenhao-db commented on PR #46123:
URL: https://github.com/apache/spark/pull/46123#issuecomment-2065464556
@HyukjinKwon could you help review? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47890][CONNECT][PYTHON] Add variant functions to Scala and Python. [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #46123:
URL: https://github.com/apache/spark/pull/46123#discussion_r1571679330
##########
python/docs/source/reference/pyspark.sql/functions.rst:
##########
@@ -533,6 +533,11 @@ JSON Functions
json_object_keys
json_tuple
parse_json
+ is_variant_null
Review Comment:
nit: let's sort the function names alphabetically
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47890][CONNECT][PYTHON] Add variant functions to Scala and Python. [spark]
Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #46123:
URL: https://github.com/apache/spark/pull/46123#discussion_r1574020712
##########
python/pyspark/sql/tests/test_functions.py:
##########
@@ -1315,6 +1315,35 @@ def test_parse_json(self):
self.assertEqual("""{"a":1}""", actual["var"])
self.assertEqual("""{"b":[{"c":"str2"}]}""", actual["var_lit"])
+ def test_variant_expressions(self):
+ df = self.spark.createDataFrame([Row(json="""{ "a" : 1 }"""), Row(json="""{ "b" : 2 }""")])
+ v = F.parse_json(df.json)
+
+ def check(resultDf, expected):
+ self.assertEqual([r[0] for r in resultDf.collect()], expected)
+
+ check(df.select(F.is_variant_null(v)), [False, False])
+ check(df.select(F.schema_of_variant(v)), ["STRUCT<a: BIGINT>", "STRUCT<b: BIGINT>"])
+ check(df.select(F.schema_of_variant_agg(v)), ["STRUCT<a: BIGINT, b: BIGINT>"])
+
+ check(df.select(F.variant_get(v, "$.a", "int")), [1, None])
+ check(df.select(F.variant_get(v, "$.b", "int")), [None, 2])
+ check(df.select(F.variant_get(v, "$.a", "double")), [1.0, None])
+
+ with self.assertRaises(SparkRuntimeException) as ex:
+ df.select(F.variant_get(v, "$.a", "binary")).collect()
Review Comment:
qq: So, only `variant_get` cast an exception right?
If we have other negative cases let's add more failure test, otherwise looks good as it is.
##########
python/pyspark/sql/tests/test_functions.py:
##########
@@ -1315,6 +1315,35 @@ def test_parse_json(self):
self.assertEqual("""{"a":1}""", actual["var"])
self.assertEqual("""{"b":[{"c":"str2"}]}""", actual["var_lit"])
+ def test_variant_expressions(self):
+ df = self.spark.createDataFrame([Row(json="""{ "a" : 1 }"""), Row(json="""{ "b" : 2 }""")])
+ v = F.parse_json(df.json)
+
+ def check(resultDf, expected):
+ self.assertEqual([r[0] for r in resultDf.collect()], expected)
+
+ check(df.select(F.is_variant_null(v)), [False, False])
+ check(df.select(F.schema_of_variant(v)), ["STRUCT<a: BIGINT>", "STRUCT<b: BIGINT>"])
+ check(df.select(F.schema_of_variant_agg(v)), ["STRUCT<a: BIGINT, b: BIGINT>"])
+
+ check(df.select(F.variant_get(v, "$.a", "int")), [1, None])
+ check(df.select(F.variant_get(v, "$.b", "int")), [None, 2])
+ check(df.select(F.variant_get(v, "$.a", "double")), [1.0, None])
+
+ with self.assertRaises(SparkRuntimeException) as ex:
+ df.select(F.variant_get(v, "$.a", "binary")).collect()
Review Comment:
qq: So, only `variant_get` cast an exception, right?
If we have other negative cases let's add more failure test, otherwise looks good as it is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47890][CONNECT][PYTHON] Add variant functions to Scala and Python. [spark]
Posted by "chenhao-db (via GitHub)" <gi...@apache.org>.
chenhao-db commented on code in PR #46123:
URL: https://github.com/apache/spark/pull/46123#discussion_r1571848958
##########
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -6975,6 +6975,71 @@ object functions {
*/
def parse_json(json: Column): Column = Column.fn("parse_json", json)
+ /**
+ * Check if a variant value is a variant null. Returns true if and only if the input is a
+ * variant null and false otherwise (including in the case of SQL NULL).
+ *
+ * @param v
+ * a variant column.
+ * @group variant_funcs
Review Comment:
Added. Please take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47890][CONNECT][PYTHON] Add variant functions to Scala and Python. [spark]
Posted by "chenhao-db (via GitHub)" <gi...@apache.org>.
chenhao-db commented on code in PR #46123:
URL: https://github.com/apache/spark/pull/46123#discussion_r1571795604
##########
python/docs/source/reference/pyspark.sql/functions.rst:
##########
@@ -533,6 +533,11 @@ JSON Functions
json_object_keys
json_tuple
parse_json
+ is_variant_null
Review Comment:
Done. It looks like `functions.py/scala` is not strictly sorted/grouped, so I only make `functions.rst` sorted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47890][CONNECT][PYTHON] Add variant functions to Scala and Python. [spark]
Posted by "chenhao-db (via GitHub)" <gi...@apache.org>.
chenhao-db commented on code in PR #46123:
URL: https://github.com/apache/spark/pull/46123#discussion_r1572455657
##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/PlanGenerationTestSuite.scala:
##########
@@ -2485,6 +2485,30 @@ class PlanGenerationTestSuite
fn.to_json(fn.col("d"), Map(("timestampFormat", "dd/MM/yyyy")))
}
+ functionTest("parse_json") {
Review Comment:
Done. Thanks for pointing out!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47890][CONNECT][PYTHON] Add variant functions to Scala and Python. [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #46123:
URL: https://github.com/apache/spark/pull/46123#discussion_r1571817704
##########
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -6975,6 +6975,71 @@ object functions {
*/
def parse_json(json: Column): Column = Column.fn("parse_json", json)
+ /**
+ * Check if a variant value is a variant null. Returns true if and only if the input is a
+ * variant null and false otherwise (including in the case of SQL NULL).
+ *
+ * @param v
+ * a variant column.
+ * @group variant_funcs
Review Comment:
We should add relevant test cases in PlanGenerationTestSuite for `connect`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47890][CONNECT][PYTHON] Add variant functions to Scala and Python. [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #46123: [SPARK-47890][CONNECT][PYTHON] Add variant functions to Scala and Python.
URL: https://github.com/apache/spark/pull/46123
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47890][CONNECT][PYTHON] Add variant functions to Scala and Python. [spark]
Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #46123:
URL: https://github.com/apache/spark/pull/46123#discussion_r1574132243
##########
python/pyspark/sql/tests/test_functions.py:
##########
@@ -1315,6 +1315,35 @@ def test_parse_json(self):
self.assertEqual("""{"a":1}""", actual["var"])
self.assertEqual("""{"b":[{"c":"str2"}]}""", actual["var_lit"])
+ def test_variant_expressions(self):
+ df = self.spark.createDataFrame([Row(json="""{ "a" : 1 }"""), Row(json="""{ "b" : 2 }""")])
+ v = F.parse_json(df.json)
+
+ def check(resultDf, expected):
+ self.assertEqual([r[0] for r in resultDf.collect()], expected)
+
+ check(df.select(F.is_variant_null(v)), [False, False])
+ check(df.select(F.schema_of_variant(v)), ["STRUCT<a: BIGINT>", "STRUCT<b: BIGINT>"])
+ check(df.select(F.schema_of_variant_agg(v)), ["STRUCT<a: BIGINT, b: BIGINT>"])
+
+ check(df.select(F.variant_get(v, "$.a", "int")), [1, None])
+ check(df.select(F.variant_get(v, "$.b", "int")), [None, 2])
+ check(df.select(F.variant_get(v, "$.a", "double")), [1.0, None])
+
+ with self.assertRaises(SparkRuntimeException) as ex:
+ df.select(F.variant_get(v, "$.a", "binary")).collect()
Review Comment:
Got it. Looks good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47890][CONNECT][PYTHON] Add variant functions to Scala and Python. [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #46123:
URL: https://github.com/apache/spark/pull/46123#issuecomment-2068681257
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47890][CONNECT][PYTHON] Add variant functions to Scala and Python. [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #46123:
URL: https://github.com/apache/spark/pull/46123#discussion_r1574083962
##########
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -6975,6 +6975,71 @@ object functions {
*/
def parse_json(json: Column): Column = Column.fn("parse_json", json)
+ /**
+ * Check if a variant value is a variant null. Returns true if and only if the input is a
+ * variant null and false otherwise (including in the case of SQL NULL).
+ *
+ * @param v
+ * a variant column.
+ * @group variant_funcs
Review Comment:
@chenhao-db I mean group `variant_funcs` is missing [here](https://github.com/apache/spark/blob/9f34b8eca2f3578367352ec28532e29fe3a91864/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/functions.scala#L58-L81)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47890][CONNECT][PYTHON] Add variant functions to Scala and Python. [spark]
Posted by "chenhao-db (via GitHub)" <gi...@apache.org>.
chenhao-db commented on code in PR #46123:
URL: https://github.com/apache/spark/pull/46123#discussion_r1571796823
##########
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -6975,6 +6975,71 @@ object functions {
*/
def parse_json(json: Column): Column = Column.fn("parse_json", json)
+ /**
+ * Check if a variant value is a variant null. Returns true if and only if the input is a
+ * variant null and false otherwise (including in the case of SQL NULL).
+ *
+ * @param v
+ * a variant column.
+ * @group variant_funcs
Review Comment:
Actually, we do have this group: https://github.com/apache/spark/blob/fe47edece059e9189d8500b3c9b3881b44678785/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L54. `parse_json` in `functions.py/scala` was incorrectly marked `json_funcs`, I also fix it by the way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47890][CONNECT][PYTHON] Add variant functions to Scala and Python. [spark]
Posted by "chenhao-db (via GitHub)" <gi...@apache.org>.
chenhao-db commented on PR #46123:
URL: https://github.com/apache/spark/pull/46123#issuecomment-2065768058
@zhengruifeng Thanks! Please take another look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org