You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhengruifeng (via GitHub)" <gi...@apache.org> on 2023/07/26 00:58:20 UTC
[GitHub] [spark] zhengruifeng commented on a diff in pull request #42151: [WIP][PYTHON][DOCS] Refine the docs for `Union`, `UnionAll` and `unionByName`
zhengruifeng commented on code in PR #42151:
URL: https://github.com/apache/spark/pull/42151#discussion_r1274257724
##########
python/pyspark/sql/dataframe.py:
##########
@@ -3831,42 +3856,57 @@ def unionByName(self, other: "DataFrame", allowMissingColumns: bool = False) ->
allowMissingColumns : bool, optional, default False
Specify whether to allow missing columns.
- .. versionadded:: 3.1.0
-
Returns
-------
:class:`DataFrame`
- Combined DataFrame.
+ A new :class:`DataFrame` containing the combined rows with corresponding
+ columns of the two given DataFrames.
+
+ Raises
+ ------
+ Py4JJavaError : If the column name conflict
Examples
--------
- The difference between this function and :func:`union` is that this function
- resolves columns by name (not by position):
-
+ Example 1: Union of two DataFrames by Name
>>> df1 = spark.createDataFrame([[1, 2, 3]], ["col0", "col1", "col2"])
>>> df2 = spark.createDataFrame([[4, 5, 6]], ["col1", "col2", "col0"])
- >>> df1.unionByName(df2).show()
+ >>> df3 = df1.unionByName(df2)
+ >>> df3.show()
+----+----+----+
|col0|col1|col2|
+----+----+----+
| 1| 2| 3|
| 6| 4| 5|
+----+----+----+
- When the parameter `allowMissingColumns` is ``True``, the set of column names
- in this and other :class:`DataFrame` can differ; missing columns will be filled with null.
- Further, the missing columns of this :class:`DataFrame` will be added at the end
- in the schema of the union result:
-
+ Example 2: Union of DataFrames with missing columns
>>> df1 = spark.createDataFrame([[1, 2, 3]], ["col0", "col1", "col2"])
>>> df2 = spark.createDataFrame([[4, 5, 6]], ["col1", "col2", "col3"])
- >>> df1.unionByName(df2, allowMissingColumns=True).show()
+ >>> df3 = df1.unionByName(df2, allowMissingColumns=True)
+ >>> df3.show()
+----+----+----+----+
|col0|col1|col2|col3|
+----+----+----+----+
- | 1| 2| 3|NULL|
- |NULL| 4| 5| 6|
+ | 1| 2| 3|null|
+ |null| 4| 5| 6|
Review Comment:
no, I just copy-paste the result from LLM.
And the new example result is wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org