You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "LuciferYang (via GitHub)" <gi...@apache.org> on 2023/09/27 09:34:49 UTC
[GitHub] [spark] LuciferYang opened a new pull request, #43145: [SPARK-45353][PYTHON][DOCS] Refine docstring of `create_map/slice/array_join`
LuciferYang opened a new pull request, #43145:
URL: https://github.com/apache/spark/pull/43145
### What changes were proposed in this pull request?
This pr refine docstring of `create_map/slice/array_join` and add some new examples.
### Why are the changes needed?
To improve PySpark documentation
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Pass Github Actions
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #43145: [SPARK-45353][PYTHON][DOCS] Refine docstring of `create_map/slice/array_join`
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #43145:
URL: https://github.com/apache/spark/pull/43145#discussion_r1338529056
##########
python/pyspark/sql/functions.py:
##########
@@ -12052,24 +12132,68 @@ def array_join(
Parameters
----------
col : :class:`~pyspark.sql.Column` or str
- target column to work on.
+ The input column containing the arrays to be joined.
delimiter : str
- delimiter used to concatenate elements
+ The string to be used as the delimiter when joining the array elements.
null_replacement : str, optional
- if set then null values will be replaced by this value
+ The string to replace null values within the array. If not set, null values are ignored.
Returns
-------
:class:`~pyspark.sql.Column`
- a column of string type. Concatenated values.
+ A new column of string type, where each value is the result of joining the corresponding
+ array from the input column.
Examples
--------
- >>> df = spark.createDataFrame([(["a", "b", "c"],), (["a", None],)], ['data'])
- >>> df.select(array_join(df.data, ",").alias("joined")).collect()
- [Row(joined='a,b,c'), Row(joined='a')]
- >>> df.select(array_join(df.data, ",", "NULL").alias("joined")).collect()
- [Row(joined='a,b,c'), Row(joined='a,NULL')]
+ Example 1: Basic usage of array_join function.
+
+ >>> from pyspark.sql import functions as sf
+ >>> df = spark.createDataFrame([(["a", "b", "c"],), (["a", "b"],)], ['data'])
+ >>> df.select(sf.array_join(df.data, ",").alias("joined")).show()
Review Comment:
let's remove all the alias in this PR, if possible
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #43145: [SPARK-45353][PYTHON][DOCS] Refine docstring of `create_map/slice/array_join`
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #43145:
URL: https://github.com/apache/spark/pull/43145#discussion_r1338527370
##########
python/pyspark/sql/functions.py:
##########
@@ -12052,24 +12132,68 @@ def array_join(
Parameters
----------
col : :class:`~pyspark.sql.Column` or str
- target column to work on.
+ The input column containing the arrays to be joined.
delimiter : str
- delimiter used to concatenate elements
+ The string to be used as the delimiter when joining the array elements.
null_replacement : str, optional
- if set then null values will be replaced by this value
+ The string to replace null values within the array. If not set, null values are ignored.
Returns
-------
:class:`~pyspark.sql.Column`
- a column of string type. Concatenated values.
+ A new column of string type, where each value is the result of joining the corresponding
+ array from the input column.
Examples
--------
- >>> df = spark.createDataFrame([(["a", "b", "c"],), (["a", None],)], ['data'])
- >>> df.select(array_join(df.data, ",").alias("joined")).collect()
- [Row(joined='a,b,c'), Row(joined='a')]
- >>> df.select(array_join(df.data, ",", "NULL").alias("joined")).collect()
- [Row(joined='a,b,c'), Row(joined='a,NULL')]
+ Example 1: Basic usage of array_join function.
+
+ >>> from pyspark.sql import functions as sf
+ >>> df = spark.createDataFrame([(["a", "b", "c"],), (["a", "b"],)], ['data'])
+ >>> df.select(sf.array_join(df.data, ",").alias("joined")).show()
+ +------+
+ |joined|
+ +------+
+ | a,b,c|
+ | a,b|
+ +------+
+
+ Example 2: Usage of array_join function with null_replacement argument.
+
+ >>> from pyspark.sql import functions as sf
+ >>> df = spark.createDataFrame([(["a", None, "c"],)], ['data'])
+ >>> df.select(sf.array_join(df.data, ",", "NULL").alias("joined")).show()
Review Comment:
to show the nullabilty, what about adding an new example to show NULL will be skipped in
```
In [10]: df.select(sf.array_join(df.data, ",").alias("joined")).show()
+------+
|joined|
+------+
| a,c|
+------+
```
##########
python/pyspark/sql/functions.py:
##########
@@ -12052,24 +12132,68 @@ def array_join(
Parameters
----------
col : :class:`~pyspark.sql.Column` or str
- target column to work on.
+ The input column containing the arrays to be joined.
delimiter : str
- delimiter used to concatenate elements
+ The string to be used as the delimiter when joining the array elements.
null_replacement : str, optional
- if set then null values will be replaced by this value
+ The string to replace null values within the array. If not set, null values are ignored.
Returns
-------
:class:`~pyspark.sql.Column`
- a column of string type. Concatenated values.
+ A new column of string type, where each value is the result of joining the corresponding
+ array from the input column.
Examples
--------
- >>> df = spark.createDataFrame([(["a", "b", "c"],), (["a", None],)], ['data'])
- >>> df.select(array_join(df.data, ",").alias("joined")).collect()
- [Row(joined='a,b,c'), Row(joined='a')]
- >>> df.select(array_join(df.data, ",", "NULL").alias("joined")).collect()
- [Row(joined='a,b,c'), Row(joined='a,NULL')]
+ Example 1: Basic usage of array_join function.
+
+ >>> from pyspark.sql import functions as sf
+ >>> df = spark.createDataFrame([(["a", "b", "c"],), (["a", "b"],)], ['data'])
+ >>> df.select(sf.array_join(df.data, ",").alias("joined")).show()
Review Comment:
What about removing such alias `.alias("joined")`? so that we can also validate the default column name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #43145: [SPARK-45353][PYTHON][DOCS] Refine docstring of `create_map/slice/array_join`
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #43145:
URL: https://github.com/apache/spark/pull/43145#issuecomment-1738369597
Merged into master for Spark 4.0. Thanks @zhengruifeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang closed pull request #43145: [SPARK-45353][PYTHON][DOCS] Refine docstring of `create_map/slice/array_join`
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang closed pull request #43145: [SPARK-45353][PYTHON][DOCS] Refine docstring of `create_map/slice/array_join`
URL: https://github.com/apache/spark/pull/43145
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org