[GitHub] [spark] HyukjinKwon commented on a change in pull request #29657: [SPARK-32798][PYTHON] Make unionByName optionally fill missing columns with nulls in PySpark

[GitBox <gi...@apache.org>]

HyukjinKwon commented on a change in pull request #29657:
URL: https://github.com/apache/spark/pull/29657#discussion_r484228920



##########
File path: python/pyspark/sql/dataframe.py
##########
@@ -1567,8 +1567,25 @@ def unionByName(self, other):
         |   1|   2|   3|
         |   6|   4|   5|
         +----+----+----+
+
+        If you want to allow missing columns, set `allowMissingColumns` to ``True``.

Review comment:
       Let's reuse the doc in Scala side.
   
   https://github.com/apache/spark/blob/84efa04c57a95ee2d289fa699036f1aa44840f72/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L2041-L2044
   
   
   ```
   When the parameter `allowMissingColumns` is true, this function allows different set of column names between 
   two :class:`DataFrame`\s. Missing columns at each side, will be filled with null values. The missing columns at left 
   :class:`DataFrame` will be added at the end in the schema of the union result.
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org