You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "sandip-db (via GitHub)" <gi...@apache.org> on 2023/09/15 04:53:30 UTC
[GitHub] [spark] sandip-db opened a new pull request, #42938: [SPARK-44788][CONNECT][PYTHON][SQL] Add from_xml and schema_of_xml to pyspark, spark connect and sql function
sandip-db opened a new pull request, #42938:
URL: https://github.com/apache/spark/pull/42938
### What changes were proposed in this pull request?
Add from_xml and schema_of_xml to pyspark, spark connect and sql function
### Why are the changes needed?
from_xml parses XML data nested in a `Column` into a struct. schema_of_xml infers schema from XML data in a `Column`.
This PR adds these two functions to pyspark, spark connect and SQL function registry.
It is one of the series of PR to add native support for [XML File Format](https://issues.apache.org/jira/browse/SPARK-44265) in spark.
### Does this PR introduce _any_ user-facing change?
Yes, it adds from_xml and schema_of_xml to pyspark, spark connect and sql function
### How was this patch tested?
- Added new unit tests
- Github Action
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #42938: [SPARK-44788][CONNECT][PYTHON][SQL] Add from_xml and schema_of_xml to pyspark, spark connect and sql function
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #42938:
URL: https://github.com/apache/spark/pull/42938#issuecomment-1722889128
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #42938: [SPARK-44788][CONNECT][PYTHON][SQL] Add from_xml and schema_of_xml to pyspark, spark connect and sql function
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #42938:
URL: https://github.com/apache/spark/pull/42938#discussion_r1326888870
##########
python/pyspark/sql/functions.py:
##########
@@ -13041,6 +13041,120 @@ def json_object_keys(col: "ColumnOrName") -> Column:
return _invoke_function_over_columns("json_object_keys", col)
+@_try_remote_functions
+def from_xml(
+ col: "ColumnOrName",
+ schema: Union[StructType, Column, str],
+ options: Optional[Dict[str, str]] = None,
+) -> Column:
+ """
+ Parses a column containing a XML string to a row with
+ the specified schema. Returns `null`, in the case of an unparseable string.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ col : :class:`~pyspark.sql.Column` or str
+ a column or column name in XML format
+ schema : :class:`StructType` or str
+ a StructType or Python string literal with a DDL-formatted string
+ to use when parsing the Xml column
+ options : dict, optional
+ options to control parsing. accepts the same options as the Xml datasource.
+ See `Data Source Option <https://spark.apache.org/docs/latest/sql-data-sources-xml.html#data-source-option>`_
+ for the version you use.
+
+ .. # noqa
+
+ Returns
+ -------
+ :class:`~pyspark.sql.Column`
+ a new column of complex type from given XML object.
+
+ Examples
+ --------
+ >>> from pyspark.sql.types import *
+ >>> from pyspark.sql.functions import from_xml, schema_of_xml, lit
+ >>> data = [(1, '''<p><a>1</a></p>''')]
+ >>> schema = StructType([StructField("a", IntegerType())])
+ >>> df = spark.createDataFrame(data, ("key", "value"))
+ >>> df.select(from_xml(df.value, schema).alias("xml")).collect()
+ [Row(xml=Row(a=1))]
+ >>> df.select(from_xml(df.value, "a INT").alias("xml")).collect()
+ [Row(xml=Row(a=1))]
+ >>> data = [(1, '<p><a>1</a><a>2</a></p>')]
+ >>> df = spark.createDataFrame(data, ("key", "value"))
+ >>> schema = StructType([StructField("a", ArrayType(IntegerType()))])
+ >>> df.select(from_xml(df.value, schema).alias("xml")).collect()
+ [Row(xml=Row(a=[1, 2]))]
+ >>> schema = schema_of_xml(lit(data[0][1]))
+ >>> df.select(from_xml(df.value, schema).alias("xml")).collect()
+ [Row(xml=Row(a=[1, 2]))]
+ """
+
+ if isinstance(schema, StructType):
+ schema = schema.json()
+ elif isinstance(schema, Column):
+ schema = _to_java_column(schema)
+ elif not isinstance(schema, str):
+ raise PySparkTypeError(
+ error_class="NOT_COLUMN_OR_STR_OR_STRUCT",
+ message_parameters={"arg_name": "schema", "arg_type": type(schema).__name__},
+ )
+ return _invoke_function("from_xml", _to_java_column(col), schema, _options_to_str(options))
+
+
+@_try_remote_functions
+def schema_of_xml(xml: "ColumnOrName", options: Optional[Dict[str, str]] = None) -> Column:
+ """
+ Parses a XML string and infers its schema in DDL format.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ xml : :class:`~pyspark.sql.Column` or str
+ a XML string or a foldable string column containing a XML string.
+ options : dict, optional
+ options to control parsing. accepts the same options as the XML datasource.
+ See `Data Source Option <https://spark.apache.org/docs/latest/sql-data-sources-xml.html#data-source-option>`_
+ for the version you use.
+
+ .. # noqa
+
+ .. versionchanged:: 4.0.0
Review Comment:
You can remove this since this is a new feature, we won't need to annotate them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #42938: [SPARK-44788][CONNECT][PYTHON][SQL] Add from_xml and schema_of_xml to pyspark, spark connect and sql function
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #42938:
URL: https://github.com/apache/spark/pull/42938#discussion_r1326888182
##########
python/pyspark/sql/functions.py:
##########
@@ -13041,6 +13041,120 @@ def json_object_keys(col: "ColumnOrName") -> Column:
return _invoke_function_over_columns("json_object_keys", col)
+@_try_remote_functions
+def from_xml(
+ col: "ColumnOrName",
+ schema: Union[StructType, Column, str],
+ options: Optional[Dict[str, str]] = None,
+) -> Column:
+ """
+ Parses a column containing a XML string to a row with
+ the specified schema. Returns `null`, in the case of an unparseable string.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ col : :class:`~pyspark.sql.Column` or str
+ a column or column name in XML format
+ schema : :class:`StructType` or str
+ a StructType or Python string literal with a DDL-formatted string
+ to use when parsing the Xml column
+ options : dict, optional
+ options to control parsing. accepts the same options as the Xml datasource.
Review Comment:
```suggestion
options to control parsing. Accepts the same options as the Xml datasource.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sandip-db commented on a diff in pull request #42938: [SPARK-44788][CONNECT][PYTHON][SQL] Add from_xml and schema_of_xml to pyspark, spark connect and sql function
Posted by "sandip-db (via GitHub)" <gi...@apache.org>.
sandip-db commented on code in PR #42938:
URL: https://github.com/apache/spark/pull/42938#discussion_r1327401538
##########
python/pyspark/sql/tests/connect/test_connect_function.py:
##########
@@ -1821,6 +1821,106 @@ def test_json_functions(self):
sdf.select(SF.to_json(SF.struct(SF.lit("a"), SF.lit("b")), {"mode": "FAILFAST"})),
)
+ def test_xml_functions(self):
+ query = """
+ SELECT * FROM VALUES
+ ('<p><a>1</a></p>', '<p><a>1</a><a>2</a><a>3</a></p>', '<p><a attr="s"><b>5.0</b></a></p>'),
+ ('<p><a>0</a></p>', '<p><a>4</a><a>5</a><a>6</a></p>', '<p><a attr="t"></a></p>')
+ AS tab(a, b, c)
+ """
+ # +---------------+-------------------------------+---------------------------------+
+ # | a| b| c|
+ # +---------------+-------------------------------+---------------------------------+
+ # |<p><a>1</a></p>|<p><a>1</a><a>2</a><a>3</a></p>|<p><a attr="s"><b>5.0</b></a></p>|
+ # |<p><a>1</a></p>|<p><a>4</a><a>5</a><a>6</a></p>| <p><a attr="t"></a></p>|
+ # +---------------+-------------------------------+---------------------------------+
+
+ cdf = self.connect.sql(query)
+ sdf = self.spark.sql(query)
+
+ # test from_xml
+ for schema in [
+ "a INT",
+# StructType([StructField("a", IntegerType())]),
Review Comment:
I would like to support this. pyspark sql works fine, but pyspark connect is failing to parse the StructType.
Any pointers would help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #42938: [SPARK-44788][CONNECT][PYTHON][SQL] Add from_xml and schema_of_xml to pyspark, spark connect and sql function
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #42938:
URL: https://github.com/apache/spark/pull/42938#discussion_r1326886753
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala:
##########
@@ -830,7 +830,11 @@ object FunctionRegistry {
// csv
expression[CsvToStructs]("from_csv"),
expression[SchemaOfCsv]("schema_of_csv"),
- expression[StructsToCsv]("to_csv")
+ expression[StructsToCsv]("to_csv"),
+
+ // Xml
+ expression[XmlToStructs] ("from_xml"),
Review Comment:
To register this properly, I think `XmlToStructs` has to be decorated by `ExpressionDescription` like you did in `SchemaOfXml`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sandip-db commented on a diff in pull request #42938: [SPARK-44788][CONNECT][PYTHON][SQL] Add from_xml and schema_of_xml to pyspark, spark connect and sql function
Posted by "sandip-db (via GitHub)" <gi...@apache.org>.
sandip-db commented on code in PR #42938:
URL: https://github.com/apache/spark/pull/42938#discussion_r1327417666
##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/FunctionTestSuite.scala:
##########
@@ -229,6 +229,18 @@ class FunctionTestSuite extends ConnectFunSuite {
schema_of_csv("x,y"),
schema_of_csv(lit("x,y"), Collections.emptyMap()))
testEquals("to_csv", to_csv(a), to_csv(a, Collections.emptyMap[String, String]))
+ testEquals(
+ "from_xml",
+ from_xml(a, schema),
+ from_xml(a, lit(schema.toDDL)),
+ // from_xml(a, lit(schema.json)),
Review Comment:
@HyukjinKwon @itholic
This is failing due to a parse error. So I commented it temporarily. The same passes for from_json() above in this test.
Can you please share some pointers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #42938: [SPARK-44788][CONNECT][PYTHON][SQL] Add from_xml and schema_of_xml to pyspark, spark connect and sql function
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #42938:
URL: https://github.com/apache/spark/pull/42938#discussion_r1328198653
##########
python/pyspark/sql/tests/connect/test_connect_function.py:
##########
@@ -1821,6 +1821,111 @@ def test_json_functions(self):
sdf.select(SF.to_json(SF.struct(SF.lit("a"), SF.lit("b")), {"mode": "FAILFAST"})),
)
+ def test_xml_functions(self):
+ query = """
+ SELECT * FROM VALUES
+ ('<p><a>1</a></p>', '<p><a>1</a><a>2</a><a>3</a></p>',
+ '<p><a attr="s"><b>5.0</b></a></p>'),
+ ('<p><a>0</a></p>', '<p><a>4</a><a>5</a><a>6</a></p>', '<p><a attr="t"></a></p>')
+ AS tab(a, b, c)
+ """
+ # +---------------+-------------------------------+---------------------------------+
+ # | a| b| c|
+ # +---------------+-------------------------------+---------------------------------+
+ # |<p><a>1</a></p>|<p><a>1</a><a>2</a><a>3</a></p>|<p><a attr="s"><b>5.0</b></a></p>|
+ # |<p><a>1</a></p>|<p><a>4</a><a>5</a><a>6</a></p>| <p><a attr="t"></a></p>|
+ # +---------------+-------------------------------+---------------------------------+
+
+ cdf = self.connect.sql(query)
+ sdf = self.spark.sql(query)
+
+ # test from_xml
+ for schema in [
+ "a INT",
+ # StructType([StructField("a", IntegerType())]),
+ # StructType([StructField("a", ArrayType(IntegerType()))]),
Review Comment:
are they supported?
##########
python/pyspark/sql/tests/connect/test_connect_function.py:
##########
@@ -1821,6 +1821,111 @@ def test_json_functions(self):
sdf.select(SF.to_json(SF.struct(SF.lit("a"), SF.lit("b")), {"mode": "FAILFAST"})),
)
+ def test_xml_functions(self):
+ query = """
+ SELECT * FROM VALUES
+ ('<p><a>1</a></p>', '<p><a>1</a><a>2</a><a>3</a></p>',
+ '<p><a attr="s"><b>5.0</b></a></p>'),
+ ('<p><a>0</a></p>', '<p><a>4</a><a>5</a><a>6</a></p>', '<p><a attr="t"></a></p>')
+ AS tab(a, b, c)
+ """
+ # +---------------+-------------------------------+---------------------------------+
+ # | a| b| c|
+ # +---------------+-------------------------------+---------------------------------+
+ # |<p><a>1</a></p>|<p><a>1</a><a>2</a><a>3</a></p>|<p><a attr="s"><b>5.0</b></a></p>|
+ # |<p><a>1</a></p>|<p><a>4</a><a>5</a><a>6</a></p>| <p><a attr="t"></a></p>|
+ # +---------------+-------------------------------+---------------------------------+
+
+ cdf = self.connect.sql(query)
+ sdf = self.spark.sql(query)
+
+ # test from_xml
+ for schema in [
+ "a INT",
+ # StructType([StructField("a", IntegerType())]),
+ # StructType([StructField("a", ArrayType(IntegerType()))]),
+ ]:
+ self.compare_by_show(
+ cdf.select(CF.from_xml(cdf.a, schema)),
+ sdf.select(SF.from_xml(sdf.a, schema)),
+ )
+ self.compare_by_show(
+ cdf.select(CF.from_xml("a", schema)),
+ sdf.select(SF.from_xml("a", schema)),
+ )
+ self.compare_by_show(
+ cdf.select(CF.from_xml(cdf.a, schema, {"mode": "FAILFAST"})),
+ sdf.select(SF.from_xml(sdf.a, schema, {"mode": "FAILFAST"})),
+ )
+ self.compare_by_show(
+ cdf.select(CF.from_xml("a", schema, {"mode": "FAILFAST"})),
+ sdf.select(SF.from_xml("a", schema, {"mode": "FAILFAST"})),
+ )
+
+ for schema in [
+ "STRUCT<a: ARRAY<INT>>",
+ # StructType([StructField("a", ArrayType(IntegerType()))]),
Review Comment:
is this supported?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a diff in pull request #42938: [SPARK-44788][CONNECT][PYTHON][SQL] Add from_xml and schema_of_xml to pyspark, spark connect and sql function
Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #42938:
URL: https://github.com/apache/spark/pull/42938#discussion_r1327925718
##########
python/pyspark/sql/tests/test_functions.py:
##########
@@ -1286,6 +1281,27 @@ def test_from_csv(self):
message_parameters={"arg_name": "schema", "arg_type": "int"},
)
+ def test_schema_of_xml(self):
Review Comment:
I think we might need to test for non-connect as well??
Seems like we only compare between connect <> non-connect from `python/pyspark/sql/tests/connect/test_connect_function.py` by using `compare_by_show `, but maybe the test for data correctness is missing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #42938: [SPARK-44788][CONNECT][PYTHON][SQL] Add from_xml and schema_of_xml to pyspark, spark connect and sql function
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #42938:
URL: https://github.com/apache/spark/pull/42938#discussion_r1326885256
##########
python/pyspark/sql/tests/connect/test_connect_function.py:
##########
@@ -1821,6 +1821,106 @@ def test_json_functions(self):
sdf.select(SF.to_json(SF.struct(SF.lit("a"), SF.lit("b")), {"mode": "FAILFAST"})),
)
+ def test_xml_functions(self):
+ query = """
+ SELECT * FROM VALUES
+ ('<p><a>1</a></p>', '<p><a>1</a><a>2</a><a>3</a></p>', '<p><a attr="s"><b>5.0</b></a></p>'),
+ ('<p><a>0</a></p>', '<p><a>4</a><a>5</a><a>6</a></p>', '<p><a attr="t"></a></p>')
+ AS tab(a, b, c)
+ """
+ # +---------------+-------------------------------+---------------------------------+
+ # | a| b| c|
+ # +---------------+-------------------------------+---------------------------------+
+ # |<p><a>1</a></p>|<p><a>1</a><a>2</a><a>3</a></p>|<p><a attr="s"><b>5.0</b></a></p>|
+ # |<p><a>1</a></p>|<p><a>4</a><a>5</a><a>6</a></p>| <p><a attr="t"></a></p>|
+ # +---------------+-------------------------------+---------------------------------+
+
+ cdf = self.connect.sql(query)
+ sdf = self.spark.sql(query)
+
+ # test from_xml
+ for schema in [
+ "a INT",
+# StructType([StructField("a", IntegerType())]),
Review Comment:
Let's probably remove commented codes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #42938: [SPARK-44788][CONNECT][PYTHON][SQL] Add from_xml and schema_of_xml to pyspark, spark connect and sql function
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #42938:
URL: https://github.com/apache/spark/pull/42938#issuecomment-1720785722
cc @itholic mind helping review this please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sandip-db commented on a diff in pull request #42938: [SPARK-44788][CONNECT][PYTHON][SQL] Add from_xml and schema_of_xml to pyspark, spark connect and sql function
Posted by "sandip-db (via GitHub)" <gi...@apache.org>.
sandip-db commented on code in PR #42938:
URL: https://github.com/apache/spark/pull/42938#discussion_r1328021810
##########
python/pyspark/sql/functions.py:
##########
@@ -13041,6 +13041,120 @@ def json_object_keys(col: "ColumnOrName") -> Column:
return _invoke_function_over_columns("json_object_keys", col)
+@_try_remote_functions
+def from_xml(
+ col: "ColumnOrName",
+ schema: Union[StructType, Column, str],
+ options: Optional[Dict[str, str]] = None,
+) -> Column:
+ """
+ Parses a column containing a XML string to a row with
+ the specified schema. Returns `null`, in the case of an unparseable string.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ col : :class:`~pyspark.sql.Column` or str
+ a column or column name in XML format
+ schema : :class:`StructType` or str
+ a StructType or Python string literal with a DDL-formatted string
+ to use when parsing the Xml column
+ options : dict, optional
+ options to control parsing. accepts the same options as the Xml datasource.
+ See `Data Source Option <https://spark.apache.org/docs/latest/sql-data-sources-xml.html#data-source-option>`_
+ for the version you use.
+
+ .. # noqa
+
+ Returns
+ -------
+ :class:`~pyspark.sql.Column`
+ a new column of complex type from given XML object.
+
+ Examples
Review Comment:
> could you please create a ticket
@allisonwang-db
Sure, created a ticket here: https://issues.apache.org/jira/browse/SPARK-45186
##########
python/pyspark/sql/functions.py:
##########
@@ -13041,6 +13041,120 @@ def json_object_keys(col: "ColumnOrName") -> Column:
return _invoke_function_over_columns("json_object_keys", col)
+@_try_remote_functions
+def from_xml(
+ col: "ColumnOrName",
+ schema: Union[StructType, Column, str],
+ options: Optional[Dict[str, str]] = None,
+) -> Column:
+ """
+ Parses a column containing a XML string to a row with
+ the specified schema. Returns `null`, in the case of an unparseable string.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ col : :class:`~pyspark.sql.Column` or str
+ a column or column name in XML format
+ schema : :class:`StructType` or str
+ a StructType or Python string literal with a DDL-formatted string
+ to use when parsing the Xml column
+ options : dict, optional
+ options to control parsing. accepts the same options as the Xml datasource.
+ See `Data Source Option <https://spark.apache.org/docs/latest/sql-data-sources-xml.html#data-source-option>`_
+ for the version you use.
+
+ .. # noqa
+
+ Returns
+ -------
+ :class:`~pyspark.sql.Column`
+ a new column of complex type from given XML object.
+
+ Examples
Review Comment:
> could you please create a ticket
@allisonwang-db
Sure, created a ticket here: https://issues.apache.org/jira/browse/SPARK-45186
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a diff in pull request #42938: [SPARK-44788][CONNECT][PYTHON][SQL] Add from_xml and schema_of_xml to pyspark, spark connect and sql function
Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #42938:
URL: https://github.com/apache/spark/pull/42938#discussion_r1327925718
##########
python/pyspark/sql/tests/test_functions.py:
##########
@@ -1286,6 +1281,27 @@ def test_from_csv(self):
message_parameters={"arg_name": "schema", "arg_type": "int"},
)
+ def test_schema_of_xml(self):
Review Comment:
I think we might need to test for non-connect as well??
Seems like we only compare between connect <> non-connect from `python/pyspark/sql/tests/connect/test_connect_function.py` by using `compare_by_show `, but maybe the test for data correctness is missing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #42938: [SPARK-44788][CONNECT][PYTHON][SQL] Add from_xml and schema_of_xml to pyspark, spark connect and sql function
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #42938:
URL: https://github.com/apache/spark/pull/42938#discussion_r1328194879
##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/FunctionTestSuite.scala:
##########
@@ -229,6 +229,18 @@ class FunctionTestSuite extends ConnectFunSuite {
schema_of_csv("x,y"),
schema_of_csv(lit("x,y"), Collections.emptyMap()))
testEquals("to_csv", to_csv(a), to_csv(a, Collections.emptyMap[String, String]))
+ testEquals(
+ "from_xml",
+ from_xml(a, schema),
+ from_xml(a, lit(schema.toDDL)),
+ // from_xml(a, lit(schema.json)),
Review Comment:
Using JSON as its schema is discouraged actually. I think you can just remove it for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #42938: [SPARK-44788][CONNECT][PYTHON][SQL] Add from_xml and schema_of_xml to pyspark, spark connect and sql function
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #42938:
URL: https://github.com/apache/spark/pull/42938#discussion_r1328213652
##########
python/pyspark/sql/tests/connect/test_connect_function.py:
##########
@@ -1821,6 +1821,111 @@ def test_json_functions(self):
sdf.select(SF.to_json(SF.struct(SF.lit("a"), SF.lit("b")), {"mode": "FAILFAST"})),
)
+ def test_xml_functions(self):
+ query = """
+ SELECT * FROM VALUES
+ ('<p><a>1</a></p>', '<p><a>1</a><a>2</a><a>3</a></p>',
+ '<p><a attr="s"><b>5.0</b></a></p>'),
+ ('<p><a>0</a></p>', '<p><a>4</a><a>5</a><a>6</a></p>', '<p><a attr="t"></a></p>')
+ AS tab(a, b, c)
+ """
+ # +---------------+-------------------------------+---------------------------------+
+ # | a| b| c|
+ # +---------------+-------------------------------+---------------------------------+
+ # |<p><a>1</a></p>|<p><a>1</a><a>2</a><a>3</a></p>|<p><a attr="s"><b>5.0</b></a></p>|
+ # |<p><a>1</a></p>|<p><a>4</a><a>5</a><a>6</a></p>| <p><a attr="t"></a></p>|
+ # +---------------+-------------------------------+---------------------------------+
+
+ cdf = self.connect.sql(query)
+ sdf = self.spark.sql(query)
+
+ # test from_xml
+ for schema in [
+ "a INT",
+ # StructType([StructField("a", IntegerType())]),
+ # StructType([StructField("a", ArrayType(IntegerType()))]),
Review Comment:
Had a quick offline sync. He will file a JIRA and take a separate look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] allisonwang-db commented on a diff in pull request #42938: [SPARK-44788][CONNECT][PYTHON][SQL] Add from_xml and schema_of_xml to pyspark, spark connect and sql function
Posted by "allisonwang-db (via GitHub)" <gi...@apache.org>.
allisonwang-db commented on code in PR #42938:
URL: https://github.com/apache/spark/pull/42938#discussion_r1327555626
##########
python/pyspark/sql/functions.py:
##########
@@ -13041,6 +13041,120 @@ def json_object_keys(col: "ColumnOrName") -> Column:
return _invoke_function_over_columns("json_object_keys", col)
+@_try_remote_functions
+def from_xml(
+ col: "ColumnOrName",
+ schema: Union[StructType, Column, str],
+ options: Optional[Dict[str, str]] = None,
+) -> Column:
+ """
+ Parses a column containing a XML string to a row with
+ the specified schema. Returns `null`, in the case of an unparseable string.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ col : :class:`~pyspark.sql.Column` or str
+ a column or column name in XML format
+ schema : :class:`StructType` or str
+ a StructType or Python string literal with a DDL-formatted string
+ to use when parsing the Xml column
+ options : dict, optional
+ options to control parsing. accepts the same options as the Xml datasource.
+ See `Data Source Option <https://spark.apache.org/docs/latest/sql-data-sources-xml.html#data-source-option>`_
+ for the version you use.
+
+ .. # noqa
+
+ Returns
+ -------
+ :class:`~pyspark.sql.Column`
+ a new column of complex type from given XML object.
+
+ Examples
Review Comment:
Documentation is extremely important for a better user-experience. @sandip-db could you please create a ticket under https://issues.apache.org/jira/browse/SPARK-44728.
##########
python/pyspark/sql/functions.py:
##########
@@ -13041,6 +13041,120 @@ def json_object_keys(col: "ColumnOrName") -> Column:
return _invoke_function_over_columns("json_object_keys", col)
+@_try_remote_functions
+def from_xml(
+ col: "ColumnOrName",
+ schema: Union[StructType, Column, str],
+ options: Optional[Dict[str, str]] = None,
+) -> Column:
+ """
+ Parses a column containing a XML string to a row with
+ the specified schema. Returns `null`, in the case of an unparseable string.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ col : :class:`~pyspark.sql.Column` or str
+ a column or column name in XML format
+ schema : :class:`StructType` or str
+ a StructType or Python string literal with a DDL-formatted string
+ to use when parsing the Xml column
+ options : dict, optional
+ options to control parsing. accepts the same options as the Xml datasource.
+ See `Data Source Option <https://spark.apache.org/docs/latest/sql-data-sources-xml.html#data-source-option>`_
+ for the version you use.
+
+ .. # noqa
+
+ Returns
+ -------
+ :class:`~pyspark.sql.Column`
+ a new column of complex type from given XML object.
+
+ Examples
Review Comment:
Documentation is extremely important for a better user-experience. @sandip-db could you please create a ticket under https://issues.apache.org/jira/browse/SPARK-44728?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #42938: [SPARK-44788][CONNECT][PYTHON][SQL] Add from_xml and schema_of_xml to pyspark, spark connect and sql function
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #42938:
URL: https://github.com/apache/spark/pull/42938#discussion_r1326881986
##########
sql/core/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -7367,15 +7367,83 @@ object functions {
* "https://spark.apache.org/docs/latest/sql-data-sources-xml.html#data-source-option">
* Data Source Option</a> in the version you use.
* @group collection_funcs
- * @since
+ * @since 4.0.0
*/
// scalastyle:on line.size.limit
def from_xml(e: Column, schema: StructType, options: Map[String, String]): Column = withExpr {
XmlToStructs(CharVarcharUtils.failIfHasCharVarchar(schema), options, e.expr)
}
+ // scalastyle:off line.size.limit
+ /**
+ * (Java-specific) Parses a column containing a XML string into a `StructType`
+ * with the specified schema.
+ * Returns `null`, in the case of an unparseable string.
+ *
+ * @param e a string column containing XML data.
+ * @param schema the schema as a DDL-formatted string.
+ * @param options options to control how the XML is parsed. accepts the same options and the
+ * xml data source.
+ * See
+ * <a href=
+ * "https://spark.apache.org/docs/latest/sql-data-sources-xml.html#data-source-option">
+ * Data Source Option</a> in the version you use.
+ * @group collection_funcs
+ * @since 4.0.0
+ */
+ // scalastyle:on line.size.limit
+ def from_xml(e: Column, schema: String, options: java.util.Map[String, String]): Column = {
+ from_xml(e, schema, options.asScala.toMap)
+ }
+
+ // scalastyle:off line.size.limit
+
+ /**
+ * (Scala-specific) Parses a column containing a XML string into a `StructType`
+ * with the specified schema.
+ * Returns `null`, in the case of an unparseable string.
+ *
+ * @param e a string column containing XML data.
+ * @param schema the schema as a DDL-formatted string.
+ * @param options options to control how the XML is parsed. accepts the same options and the
+ * Xml data source.
+ * See
+ * <a href=
+ * "https://spark.apache.org/docs/latest/sql-data-sources-xml.html#data-source-option">
+ * Data Source Option</a> in the version you use.
+ * @group collection_funcs
+ * @since 4.0.0
+ */
+ // scalastyle:on line.size.limit
+ def from_xml(e: Column, schema: String, options: Map[String, String]): Column = {
Review Comment:
Let's probably remove the Scala specific `Map`, and only have one with Java signature. Scala can easily use them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #42938: [SPARK-44788][CONNECT][PYTHON][SQL] Add from_xml and schema_of_xml to pyspark, spark connect and sql function
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #42938:
URL: https://github.com/apache/spark/pull/42938#discussion_r1326882806
##########
python/pyspark/sql/functions.py:
##########
@@ -13041,6 +13041,120 @@ def json_object_keys(col: "ColumnOrName") -> Column:
return _invoke_function_over_columns("json_object_keys", col)
+@_try_remote_functions
+def from_xml(
+ col: "ColumnOrName",
+ schema: Union[StructType, Column, str],
+ options: Optional[Dict[str, str]] = None,
+) -> Column:
+ """
+ Parses a column containing a XML string to a row with
+ the specified schema. Returns `null`, in the case of an unparseable string.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ col : :class:`~pyspark.sql.Column` or str
+ a column or column name in XML format
+ schema : :class:`StructType` or str
+ a StructType or Python string literal with a DDL-formatted string
+ to use when parsing the Xml column
+ options : dict, optional
+ options to control parsing. accepts the same options as the Xml datasource.
+ See `Data Source Option <https://spark.apache.org/docs/latest/sql-data-sources-xml.html#data-source-option>`_
+ for the version you use.
+
+ .. # noqa
+
+ Returns
+ -------
+ :class:`~pyspark.sql.Column`
+ a new column of complex type from given XML object.
+
+ Examples
Review Comment:
I think we should improve the examples here but let's do that separately. cc @allisonwang-db and @zhengruifeng FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #42938: [SPARK-44788][CONNECT][PYTHON][SQL] Add from_xml and schema_of_xml to pyspark, spark connect and sql function
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #42938:
URL: https://github.com/apache/spark/pull/42938#discussion_r1326888182
##########
python/pyspark/sql/functions.py:
##########
@@ -13041,6 +13041,120 @@ def json_object_keys(col: "ColumnOrName") -> Column:
return _invoke_function_over_columns("json_object_keys", col)
+@_try_remote_functions
+def from_xml(
+ col: "ColumnOrName",
+ schema: Union[StructType, Column, str],
+ options: Optional[Dict[str, str]] = None,
+) -> Column:
+ """
+ Parses a column containing a XML string to a row with
+ the specified schema. Returns `null`, in the case of an unparseable string.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ col : :class:`~pyspark.sql.Column` or str
+ a column or column name in XML format
+ schema : :class:`StructType` or str
+ a StructType or Python string literal with a DDL-formatted string
+ to use when parsing the Xml column
+ options : dict, optional
+ options to control parsing. accepts the same options as the Xml datasource.
Review Comment:
```suggestion
options to control parsing. Accepts the same options as the Xml datasource.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #42938: [SPARK-44788][CONNECT][PYTHON][SQL] Add from_xml and schema_of_xml to pyspark, spark connect and sql function
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #42938:
URL: https://github.com/apache/spark/pull/42938#discussion_r1326885469
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala:
##########
@@ -830,7 +830,11 @@ object FunctionRegistry {
// csv
expression[CsvToStructs]("from_csv"),
expression[SchemaOfCsv]("schema_of_csv"),
- expression[StructsToCsv]("to_csv")
+ expression[StructsToCsv]("to_csv"),
+
+ // Xml
+ expression[XmlToStructs] ("from_xml"),
Review Comment:
```suggestion
expression[XmlToStructs]("from_xml"),
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sandip-db commented on a diff in pull request #42938: [SPARK-44788][CONNECT][PYTHON][SQL] Add from_xml and schema_of_xml to pyspark, spark connect and sql function
Posted by "sandip-db (via GitHub)" <gi...@apache.org>.
sandip-db commented on code in PR #42938:
URL: https://github.com/apache/spark/pull/42938#discussion_r1328196005
##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/FunctionTestSuite.scala:
##########
@@ -229,6 +229,18 @@ class FunctionTestSuite extends ConnectFunSuite {
schema_of_csv("x,y"),
schema_of_csv(lit("x,y"), Collections.emptyMap()))
testEquals("to_csv", to_csv(a), to_csv(a, Collections.emptyMap[String, String]))
+ testEquals(
+ "from_xml",
+ from_xml(a, schema),
+ from_xml(a, lit(schema.toDDL)),
+ // from_xml(a, lit(schema.json)),
Review Comment:
It works now :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a diff in pull request #42938: [SPARK-44788][CONNECT][PYTHON][SQL] Add from_xml and schema_of_xml to pyspark, spark connect and sql function
Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #42938:
URL: https://github.com/apache/spark/pull/42938#discussion_r1327920967
##########
python/pyspark/errors/error_classes.py:
##########
@@ -477,6 +477,11 @@
"Argument `<arg_name>` should be a Column or str, got <arg_type>."
]
},
+ "NOT_COLUMN_OR_STR_OR_STRUCT" : {
+ "message" : [
+ "Argument `<arg_name>` should be a StructType or Column or str, got <arg_type>."
Review Comment:
nit:
```diff
- "Argument `<arg_name>` should be a StructType or Column or str, got <arg_type>."
+ "Argument `<arg_name>` should be a StructType, Column or str, got <arg_type>."
```
for consistency to other error messages.
##########
python/pyspark/sql/functions.py:
##########
@@ -13041,6 +13041,117 @@ def json_object_keys(col: "ColumnOrName") -> Column:
return _invoke_function_over_columns("json_object_keys", col)
+@_try_remote_functions
+def from_xml(
+ col: "ColumnOrName",
+ schema: Union[StructType, Column, str],
+ options: Optional[Dict[str, str]] = None,
+) -> Column:
+ """
+ Parses a column containing a XML string to a row with
+ the specified schema. Returns `null`, in the case of an unparseable string.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ col : :class:`~pyspark.sql.Column` or str
+ a column or column name in XML format
+ schema : :class:`StructType` or str
+ a StructType or Python string literal with a DDL-formatted string
+ to use when parsing the Xml column
+ options : dict, optional
+ options to control parsing. accepts the same options as the Xml datasource.
Review Comment:
Xml -> XML
##########
python/pyspark/sql/functions.py:
##########
@@ -13041,6 +13041,120 @@ def json_object_keys(col: "ColumnOrName") -> Column:
return _invoke_function_over_columns("json_object_keys", col)
+@_try_remote_functions
+def from_xml(
+ col: "ColumnOrName",
+ schema: Union[StructType, Column, str],
+ options: Optional[Dict[str, str]] = None,
+) -> Column:
+ """
+ Parses a column containing a XML string to a row with
+ the specified schema. Returns `null`, in the case of an unparseable string.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ col : :class:`~pyspark.sql.Column` or str
+ a column or column name in XML format
+ schema : :class:`StructType` or str
+ a StructType or Python string literal with a DDL-formatted string
+ to use when parsing the Xml column
+ options : dict, optional
+ options to control parsing. accepts the same options as the Xml datasource.
+ See `Data Source Option <https://spark.apache.org/docs/latest/sql-data-sources-xml.html#data-source-option>`_
+ for the version you use.
+
+ .. # noqa
+
+ Returns
+ -------
+ :class:`~pyspark.sql.Column`
+ a new column of complex type from given XML object.
+
+ Examples
Review Comment:
Yeah, we can do it in separate PR, but let's simply put some newline to readability for now.
e.g.
```python
>>> from pyspark.sql.types import *
>>> from pyspark.sql.functions import from_xml, schema_of_xml, lit
StructType input with simple IntegerType.
>>> data = [(1, '''<p><a>1</a></p>''')]
>>> df = spark.createDataFrame(data, ("key", "value"))
>>> schema = StructType([StructField("a", IntegerType())])
>>> df.select(from_xml(df.value, schema).alias("xml")).collect()
[Row(xml=Row(a=1))]
String input.
>>> df.select(from_xml(df.value, "a INT").alias("xml")).collect()
[Row(xml=Row(a=1))]
StructType input with complex ArrayType.
>>> data = [(1, '<p><a>1</a><a>2</a></p>')]
>>> df = spark.createDataFrame(data, ("key", "value"))
>>> schema = StructType([StructField("a", ArrayType(IntegerType()))])
>>> df.select(from_xml(df.value, schema).alias("xml")).collect()
[Row(xml=Row(a=[1, 2]))]
Column input generated by schema_of_xml.
>>> schema = schema_of_xml(lit(data[0][1]))
>>> df.select(from_xml(df.value, schema).alias("xml")).collect()
[Row(xml=Row(a=[1, 2]))]
```
##########
python/pyspark/sql/tests/connect/test_connect_function.py:
##########
@@ -1821,6 +1821,106 @@ def test_json_functions(self):
sdf.select(SF.to_json(SF.struct(SF.lit("a"), SF.lit("b")), {"mode": "FAILFAST"})),
)
+ def test_xml_functions(self):
+ query = """
+ SELECT * FROM VALUES
+ ('<p><a>1</a></p>', '<p><a>1</a><a>2</a><a>3</a></p>', '<p><a attr="s"><b>5.0</b></a></p>'),
+ ('<p><a>0</a></p>', '<p><a>4</a><a>5</a><a>6</a></p>', '<p><a attr="t"></a></p>')
+ AS tab(a, b, c)
+ """
+ # +---------------+-------------------------------+---------------------------------+
+ # | a| b| c|
+ # +---------------+-------------------------------+---------------------------------+
+ # |<p><a>1</a></p>|<p><a>1</a><a>2</a><a>3</a></p>|<p><a attr="s"><b>5.0</b></a></p>|
+ # |<p><a>1</a></p>|<p><a>4</a><a>5</a><a>6</a></p>| <p><a attr="t"></a></p>|
+ # +---------------+-------------------------------+---------------------------------+
+
+ cdf = self.connect.sql(query)
+ sdf = self.spark.sql(query)
+
+ # test from_xml
+ for schema in [
+ "a INT",
+# StructType([StructField("a", IntegerType())]),
+# StructType([StructField("a", ArrayType(IntegerType()))]),
+ ]:
+ self.compare_by_show(
+ cdf.select(CF.from_xml(cdf.a, schema)),
+ sdf.select(SF.from_xml(sdf.a, schema)),
+ )
+ self.compare_by_show(
+ cdf.select(CF.from_xml("a", schema)),
+ sdf.select(SF.from_xml("a", schema)),
+ )
+ self.compare_by_show(
+ cdf.select(CF.from_xml(cdf.a, schema, {"mode": "FAILFAST"})),
+ sdf.select(SF.from_xml(sdf.a, schema, {"mode": "FAILFAST"})),
+ )
+ self.compare_by_show(
+ cdf.select(CF.from_xml("a", schema, {"mode": "FAILFAST"})),
+ sdf.select(SF.from_xml("a", schema, {"mode": "FAILFAST"})),
+ )
+
+ for schema in [
+ "STRUCT<a: ARRAY<INT>>",
+# StructType([StructField("a", ArrayType(IntegerType()))]),
Review Comment:
ditto. And let's move the `#` for the proper place.
```diff
- # StructType([StructField("a", ArrayType(IntegerType()))]),
+ # StructType([StructField("a", ArrayType(IntegerType()))]),
```
##########
python/pyspark/sql/tests/test_functions.py:
##########
@@ -1286,6 +1281,27 @@ def test_from_csv(self):
message_parameters={"arg_name": "schema", "arg_type": "int"},
)
+ def test_schema_of_xml(self):
Review Comment:
I think we might need to test for non-connect as well??
Seems like we only compare between connect <> non-connect from `python/pyspark/sql/tests/connect/test_connect_function.py` by using `compare_by_show `, but there is no test for data correctness.
##########
python/pyspark/sql/tests/connect/test_connect_function.py:
##########
@@ -1821,6 +1821,106 @@ def test_json_functions(self):
sdf.select(SF.to_json(SF.struct(SF.lit("a"), SF.lit("b")), {"mode": "FAILFAST"})),
)
+ def test_xml_functions(self):
+ query = """
+ SELECT * FROM VALUES
+ ('<p><a>1</a></p>', '<p><a>1</a><a>2</a><a>3</a></p>', '<p><a attr="s"><b>5.0</b></a></p>'),
+ ('<p><a>0</a></p>', '<p><a>4</a><a>5</a><a>6</a></p>', '<p><a attr="t"></a></p>')
+ AS tab(a, b, c)
+ """
+ # +---------------+-------------------------------+---------------------------------+
+ # | a| b| c|
+ # +---------------+-------------------------------+---------------------------------+
+ # |<p><a>1</a></p>|<p><a>1</a><a>2</a><a>3</a></p>|<p><a attr="s"><b>5.0</b></a></p>|
+ # |<p><a>1</a></p>|<p><a>4</a><a>5</a><a>6</a></p>| <p><a attr="t"></a></p>|
+ # +---------------+-------------------------------+---------------------------------+
+
+ cdf = self.connect.sql(query)
+ sdf = self.spark.sql(query)
+
+ # test from_xml
+ for schema in [
+ "a INT",
+# StructType([StructField("a", IntegerType())]),
Review Comment:
Then let's create JIRA and comment it as TODO for now. I think we need separate investigation.
e.g.
```python
# TODO(SPARK-XXXXX): description (Please fill the actual description and JIRA number)
for schema in [
"a INT",
# StructType([StructField("a", IntegerType())]),
# StructType([StructField("a", ArrayType(IntegerType()))]),
]:
```
##########
python/pyspark/sql/functions.py:
##########
@@ -13041,6 +13041,117 @@ def json_object_keys(col: "ColumnOrName") -> Column:
return _invoke_function_over_columns("json_object_keys", col)
+@_try_remote_functions
+def from_xml(
+ col: "ColumnOrName",
+ schema: Union[StructType, Column, str],
+ options: Optional[Dict[str, str]] = None,
+) -> Column:
+ """
+ Parses a column containing a XML string to a row with
+ the specified schema. Returns `null`, in the case of an unparseable string.
+
+ .. versionadded:: 4.0.0
+
+ Parameters
+ ----------
+ col : :class:`~pyspark.sql.Column` or str
+ a column or column name in XML format
+ schema : :class:`StructType` or str
Review Comment:
```diff
- schema : :class:`StructType` or str
+ schema : :class:`StructType`, :class:`~pyspark.sql.Column` or str
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a diff in pull request #42938: [SPARK-44788][CONNECT][PYTHON][SQL] Add from_xml and schema_of_xml to pyspark, spark connect and sql function
Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #42938:
URL: https://github.com/apache/spark/pull/42938#discussion_r1327925718
##########
python/pyspark/sql/tests/test_functions.py:
##########
@@ -1286,6 +1281,27 @@ def test_from_csv(self):
message_parameters={"arg_name": "schema", "arg_type": "int"},
)
+ def test_schema_of_xml(self):
Review Comment:
I think we might need to test for non-connect as well??
Seems like we only compare between connect <> non-connect from `python/pyspark/sql/tests/connect/test_connect_function.py` by using `compare_by_show `, but maybe the test for data correctness is missing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #42938: [SPARK-44788][CONNECT][PYTHON][SQL] Add from_xml and schema_of_xml to pyspark, spark connect and sql function
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #42938: [SPARK-44788][CONNECT][PYTHON][SQL] Add from_xml and schema_of_xml to pyspark, spark connect and sql function
URL: https://github.com/apache/spark/pull/42938
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sandip-db commented on a diff in pull request #42938: [SPARK-44788][CONNECT][PYTHON][SQL] Add from_xml and schema_of_xml to pyspark, spark connect and sql function
Posted by "sandip-db (via GitHub)" <gi...@apache.org>.
sandip-db commented on code in PR #42938:
URL: https://github.com/apache/spark/pull/42938#discussion_r1328227219
##########
python/pyspark/sql/tests/connect/test_connect_function.py:
##########
@@ -1821,6 +1821,111 @@ def test_json_functions(self):
sdf.select(SF.to_json(SF.struct(SF.lit("a"), SF.lit("b")), {"mode": "FAILFAST"})),
)
+ def test_xml_functions(self):
+ query = """
+ SELECT * FROM VALUES
+ ('<p><a>1</a></p>', '<p><a>1</a><a>2</a><a>3</a></p>',
+ '<p><a attr="s"><b>5.0</b></a></p>'),
+ ('<p><a>0</a></p>', '<p><a>4</a><a>5</a><a>6</a></p>', '<p><a attr="t"></a></p>')
+ AS tab(a, b, c)
+ """
+ # +---------------+-------------------------------+---------------------------------+
+ # | a| b| c|
+ # +---------------+-------------------------------+---------------------------------+
+ # |<p><a>1</a></p>|<p><a>1</a><a>2</a><a>3</a></p>|<p><a attr="s"><b>5.0</b></a></p>|
+ # |<p><a>1</a></p>|<p><a>4</a><a>5</a><a>6</a></p>| <p><a attr="t"></a></p>|
+ # +---------------+-------------------------------+---------------------------------+
+
+ cdf = self.connect.sql(query)
+ sdf = self.spark.sql(query)
+
+ # test from_xml
+ for schema in [
+ "a INT",
+ # StructType([StructField("a", IntegerType())]),
+ # StructType([StructField("a", ArrayType(IntegerType()))]),
Review Comment:
Here is the jira:
https://issues.apache.org/jira/browse/SPARK-45190
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org