You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhengruifeng (via GitHub)" <gi...@apache.org> on 2023/09/27 12:28:36 UTC
[GitHub] [spark] zhengruifeng commented on a diff in pull request #43145: [SPARK-45353][PYTHON][DOCS] Refine docstring of `create_map/slice/array_join`
zhengruifeng commented on code in PR #43145:
URL: https://github.com/apache/spark/pull/43145#discussion_r1338527370
##########
python/pyspark/sql/functions.py:
##########
@@ -12052,24 +12132,68 @@ def array_join(
Parameters
----------
col : :class:`~pyspark.sql.Column` or str
- target column to work on.
+ The input column containing the arrays to be joined.
delimiter : str
- delimiter used to concatenate elements
+ The string to be used as the delimiter when joining the array elements.
null_replacement : str, optional
- if set then null values will be replaced by this value
+ The string to replace null values within the array. If not set, null values are ignored.
Returns
-------
:class:`~pyspark.sql.Column`
- a column of string type. Concatenated values.
+ A new column of string type, where each value is the result of joining the corresponding
+ array from the input column.
Examples
--------
- >>> df = spark.createDataFrame([(["a", "b", "c"],), (["a", None],)], ['data'])
- >>> df.select(array_join(df.data, ",").alias("joined")).collect()
- [Row(joined='a,b,c'), Row(joined='a')]
- >>> df.select(array_join(df.data, ",", "NULL").alias("joined")).collect()
- [Row(joined='a,b,c'), Row(joined='a,NULL')]
+ Example 1: Basic usage of array_join function.
+
+ >>> from pyspark.sql import functions as sf
+ >>> df = spark.createDataFrame([(["a", "b", "c"],), (["a", "b"],)], ['data'])
+ >>> df.select(sf.array_join(df.data, ",").alias("joined")).show()
+ +------+
+ |joined|
+ +------+
+ | a,b,c|
+ | a,b|
+ +------+
+
+ Example 2: Usage of array_join function with null_replacement argument.
+
+ >>> from pyspark.sql import functions as sf
+ >>> df = spark.createDataFrame([(["a", None, "c"],)], ['data'])
+ >>> df.select(sf.array_join(df.data, ",", "NULL").alias("joined")).show()
Review Comment:
to show the nullabilty, what about adding an new example to show NULL will be skipped in
```
In [10]: df.select(sf.array_join(df.data, ",").alias("joined")).show()
+------+
|joined|
+------+
| a,c|
+------+
```
##########
python/pyspark/sql/functions.py:
##########
@@ -12052,24 +12132,68 @@ def array_join(
Parameters
----------
col : :class:`~pyspark.sql.Column` or str
- target column to work on.
+ The input column containing the arrays to be joined.
delimiter : str
- delimiter used to concatenate elements
+ The string to be used as the delimiter when joining the array elements.
null_replacement : str, optional
- if set then null values will be replaced by this value
+ The string to replace null values within the array. If not set, null values are ignored.
Returns
-------
:class:`~pyspark.sql.Column`
- a column of string type. Concatenated values.
+ A new column of string type, where each value is the result of joining the corresponding
+ array from the input column.
Examples
--------
- >>> df = spark.createDataFrame([(["a", "b", "c"],), (["a", None],)], ['data'])
- >>> df.select(array_join(df.data, ",").alias("joined")).collect()
- [Row(joined='a,b,c'), Row(joined='a')]
- >>> df.select(array_join(df.data, ",", "NULL").alias("joined")).collect()
- [Row(joined='a,b,c'), Row(joined='a,NULL')]
+ Example 1: Basic usage of array_join function.
+
+ >>> from pyspark.sql import functions as sf
+ >>> df = spark.createDataFrame([(["a", "b", "c"],), (["a", "b"],)], ['data'])
+ >>> df.select(sf.array_join(df.data, ",").alias("joined")).show()
Review Comment:
What about removing such alias `.alias("joined")`? so that we can also validate the default column name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org