You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "gene-db (via GitHub)" <gi...@apache.org> on 2024/03/28 00:26:34 UTC
[PR] [SPARK-47366] Add pyspark and dataframe parse_json aliases [spark]
gene-db opened a new pull request, #45741:
URL: https://github.com/apache/spark/pull/45741
### What changes were proposed in this pull request?
Added the `parse_json` function alias for pyspark and dataframe APIs.
### Why are the changes needed?
Improves usability of the `parse_json` function.
### Does this PR introduce _any_ user-facing change?
Before this change, the following would not be possible:
```
df.select(parse_json(df.json)).collect()
```
### How was this patch tested?
Added unit tests and manual testing.
### Was this patch authored or co-authored using generative AI tooling?
no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47366][PYTHON] Add pyspark and dataframe parse_json aliases [spark]
Posted by "gene-db (via GitHub)" <gi...@apache.org>.
gene-db commented on code in PR #45741:
URL: https://github.com/apache/spark/pull/45741#discussion_r1542249535
##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -15098,6 +15098,38 @@ def from_json(
return _invoke_function("from_json", _to_java_column(col), schema, _options_to_str(options))
+@_try_remote_functions
+def parse_json(
+ col: "ColumnOrName",
+) -> Column:
+ """
+ Parses a column containing a JSON string into a :class:`VariantType`.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ col : :class:`~pyspark.sql.Column` or str
+ a column or column name JSON formatted strings
+
+ .. # noqa
+
+ Returns
+ -------
+ :class:`~pyspark.sql.Column`
+ a new column of VariantType.
+
+ Examples
+ --------
+ >>> from pyspark.sql.types import *
Review Comment:
removed.
##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -15098,6 +15098,38 @@ def from_json(
return _invoke_function("from_json", _to_java_column(col), schema, _options_to_str(options))
+@_try_remote_functions
+def parse_json(
Review Comment:
done.
##########
sql/core/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -6594,6 +6594,24 @@ object functions {
fnWithOptions("from_json", options, e, schema)
}
+ /**
+ * Parses a JSON string and constructs a Variant value.
+ *
+ * @param json a JSON string.
+ *
+ * @since 4.0.0
+ */
+ def parse_json(json: String): Column = parse_json(lit(json))
Review Comment:
removed.
##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -15098,6 +15098,38 @@ def from_json(
return _invoke_function("from_json", _to_java_column(col), schema, _options_to_str(options))
+@_try_remote_functions
+def parse_json(
+ col: "ColumnOrName",
+) -> Column:
+ """
+ Parses a column containing a JSON string into a :class:`VariantType`.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ col : :class:`~pyspark.sql.Column` or str
+ a column or column name JSON formatted strings
+
+ .. # noqa
+
+ Returns
+ -------
+ :class:`~pyspark.sql.Column`
+ a new column of VariantType.
+
+ Examples
+ --------
+ >>> from pyspark.sql.types import *
+ >>> df = spark.createDataFrame([ {'json': '''{ "a" : 1 }'''} ])
+ >>> df.select(to_json(parse_json(df.json)).alias("v")).collect()
Review Comment:
Removed alias.
##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -15098,6 +15098,38 @@ def from_json(
return _invoke_function("from_json", _to_java_column(col), schema, _options_to_str(options))
+@_try_remote_functions
+def parse_json(
+ col: "ColumnOrName",
+) -> Column:
+ """
+ Parses a column containing a JSON string into a :class:`VariantType`.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ col : :class:`~pyspark.sql.Column` or str
+ a column or column name JSON formatted strings
+
+ .. # noqa
Review Comment:
removed this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47366][PYTHON] Add pyspark and dataframe parse_json aliases [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #45741:
URL: https://github.com/apache/spark/pull/45741#discussion_r1542190216
##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -15098,6 +15098,38 @@ def from_json(
return _invoke_function("from_json", _to_java_column(col), schema, _options_to_str(options))
+@_try_remote_functions
+def parse_json(
+ col: "ColumnOrName",
+) -> Column:
+ """
+ Parses a column containing a JSON string into a :class:`VariantType`.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ col : :class:`~pyspark.sql.Column` or str
+ a column or column name JSON formatted strings
+
+ .. # noqa
+
+ Returns
+ -------
+ :class:`~pyspark.sql.Column`
+ a new column of VariantType.
+
+ Examples
+ --------
+ >>> from pyspark.sql.types import *
Review Comment:
this too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47366][PYTHON] Add pyspark and dataframe parse_json aliases [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #45741:
URL: https://github.com/apache/spark/pull/45741#issuecomment-2024504624
seems like we should add the implementation for Scala side as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47366][PYTHON] Add pyspark and dataframe parse_json aliases [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #45741:
URL: https://github.com/apache/spark/pull/45741#discussion_r1542190297
##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -15098,6 +15098,38 @@ def from_json(
return _invoke_function("from_json", _to_java_column(col), schema, _options_to_str(options))
+@_try_remote_functions
+def parse_json(
+ col: "ColumnOrName",
+) -> Column:
+ """
+ Parses a column containing a JSON string into a :class:`VariantType`.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ col : :class:`~pyspark.sql.Column` or str
+ a column or column name JSON formatted strings
+
+ .. # noqa
+
+ Returns
+ -------
+ :class:`~pyspark.sql.Column`
+ a new column of VariantType.
+
+ Examples
+ --------
+ >>> from pyspark.sql.types import *
+ >>> df = spark.createDataFrame([ {'json': '''{ "a" : 1 }'''} ])
+ >>> df.select(to_json(parse_json(df.json)).alias("v")).collect()
Review Comment:
Let's probably use `show()`. instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47366][PYTHON] Add pyspark and dataframe parse_json aliases [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #45741:
URL: https://github.com/apache/spark/pull/45741#discussion_r1542190834
##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -15098,6 +15098,38 @@ def from_json(
return _invoke_function("from_json", _to_java_column(col), schema, _options_to_str(options))
+@_try_remote_functions
+def parse_json(
Review Comment:
should be listed in `python/docs/source/reference/pyspark.sql/functions.rst`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47366][PYTHON] Add pyspark and dataframe parse_json aliases [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #45741:
URL: https://github.com/apache/spark/pull/45741#issuecomment-2026332334
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47366][PYTHON] Add pyspark and dataframe parse_json aliases [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #45741:
URL: https://github.com/apache/spark/pull/45741#discussion_r1542190658
##########
sql/core/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -6594,6 +6594,24 @@ object functions {
fnWithOptions("from_json", options, e, schema)
}
+ /**
+ * Parses a JSON string and constructs a Variant value.
+ *
+ * @param json a JSON string.
+ *
+ * @since 4.0.0
+ */
+ def parse_json(json: String): Column = parse_json(lit(json))
Review Comment:
I would remove this String type signature
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47366][PYTHON] Add pyspark and dataframe parse_json aliases [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #45741: [SPARK-47366][PYTHON] Add pyspark and dataframe parse_json aliases
URL: https://github.com/apache/spark/pull/45741
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47366][PYTHON] Add pyspark and dataframe parse_json aliases [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #45741:
URL: https://github.com/apache/spark/pull/45741#discussion_r1542190297
##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -15098,6 +15098,38 @@ def from_json(
return _invoke_function("from_json", _to_java_column(col), schema, _options_to_str(options))
+@_try_remote_functions
+def parse_json(
+ col: "ColumnOrName",
+) -> Column:
+ """
+ Parses a column containing a JSON string into a :class:`VariantType`.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ col : :class:`~pyspark.sql.Column` or str
+ a column or column name JSON formatted strings
+
+ .. # noqa
+
+ Returns
+ -------
+ :class:`~pyspark.sql.Column`
+ a new column of VariantType.
+
+ Examples
+ --------
+ >>> from pyspark.sql.types import *
+ >>> df = spark.createDataFrame([ {'json': '''{ "a" : 1 }'''} ])
+ >>> df.select(to_json(parse_json(df.json)).alias("v")).collect()
Review Comment:
Let's probably use `show()`. instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47366][PYTHON] Add pyspark and dataframe parse_json aliases [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #45741:
URL: https://github.com/apache/spark/pull/45741#discussion_r1542190154
##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -15098,6 +15098,38 @@ def from_json(
return _invoke_function("from_json", _to_java_column(col), schema, _options_to_str(options))
+@_try_remote_functions
+def parse_json(
+ col: "ColumnOrName",
+) -> Column:
+ """
+ Parses a column containing a JSON string into a :class:`VariantType`.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ col : :class:`~pyspark.sql.Column` or str
+ a column or column name JSON formatted strings
+
+ .. # noqa
Review Comment:
I think we don't need this (?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47366][PYTHON] Add pyspark and dataframe parse_json aliases [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #45741:
URL: https://github.com/apache/spark/pull/45741#discussion_r1542253499
##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -15098,6 +15098,38 @@ def from_json(
return _invoke_function("from_json", _to_java_column(col), schema, _options_to_str(options))
+@_try_remote_functions
+def parse_json(
+ col: "ColumnOrName",
+) -> Column:
+ """
+ Parses a column containing a JSON string into a :class:`VariantType`.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ col : :class:`~pyspark.sql.Column` or str
+ a column or column name JSON formatted strings
+
+ .. # noqa
+
+ Returns
+ -------
+ :class:`~pyspark.sql.Column`
+ a new column of VariantType.
+
+ Examples
+ --------
+ >>> from pyspark.sql.types import *
+ >>> df = spark.createDataFrame([ {'json': '''{ "a" : 1 }'''} ])
+ >>> df.select(to_json(parse_json(df.json)).alias("v")).collect()
Review Comment:
let's do not use alias to also verify the default column name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org