[spark] branch master updated: [SPARK-42194][PS] Allow `columns` parameter when creating DataFrame with Series

[gu...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 086c8d9d6ce [SPARK-42194][PS] Allow `columns` parameter when creating DataFrame with Series
086c8d9d6ce is described below

commit 086c8d9d6ce91974e97ab47aab1cf54974e12bbf
Author: itholic <ha...@databricks.com>
AuthorDate: Sun Jan 29 11:34:19 2023 +0900

    [SPARK-42194][PS] Allow `columns` parameter when creating DataFrame with Series
    
    ### What changes were proposed in this pull request?
    
    This PR proposes to allow `columns` parameter when creating `ps.DataFrame` with `ps.Series` with limited condition.
    
    ### Why are the changes needed?
    
    In pandas, they attach the new column consists with missing values when `columns` contains more than 2 columns including valid column:
    
    ```python
    >>> pser  # pandas Series
    0.427027    1
    0.904592    2
    0.599768    3
    Name: x, dtype: int64
    
    >>> pd.DataFrame(pser, columns=["x", "y", "z"])
              x    y    z
    0.427027  1  NaN  NaN
    0.904592  2  NaN  NaN
    0.599768  3  NaN  NaN
    ```
    
    But this method is potentially pretty expensive in pandas API on Spark, so I guess that's why we currently don't support it.
    
    However, I've seen examples of using the following:
    
    ```python
    >>> ps.DataFrame(pser, columns=["x"])
              x
    0.427027  1
    0.904592  2
    0.599768  3
    ```
    
    As shown in the example above, this just works the same as `pd.DataFrame(pser)` (without `columns`).
    
    But it fails with `ps.Series` as below:
    
    ```python
    >>> ps.DataFrame(psser, columns=["x"])  # `psser` is pandas-on-Spark Series
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File ".../spark/python/pyspark/pandas/frame.py", line 539, in __init__
        assert columns is None
    AssertionError
    ```
    
    In this case, user might just want to clearly state column names in their code, so I believe we can allow this rather than raising an `AssertionError`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    **Before**
    ```python
    >>> ps.DataFrame(psser, columns=["x"])  # `psser` is pandas-on-Spark Series
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File ".../spark/python/pyspark/pandas/frame.py", line 539, in __init__
        assert columns is None
    AssertionError
    ```
    
    **After**
    ```python
    >>> ps.DataFrame(psser, columns=["x"])  # `psser` is pandas-on-Spark Series
              x
    0.427027  1
    0.904592  2
    0.599768  3
    ```
    
    ### How was this patch tested?
    
    Added UTs.
    
    Closes #39786 from itholic/SPARK-42194.
    
    Authored-by: itholic <ha...@databricks.com>
    Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
 python/pyspark/pandas/frame.py                | 7 ++++++-
 python/pyspark/pandas/tests/test_dataframe.py | 7 +++++++
 2 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/python/pyspark/pandas/frame.py b/python/pyspark/pandas/frame.py
index a217066eff6..4a6c2119104 100644
--- a/python/pyspark/pandas/frame.py
+++ b/python/pyspark/pandas/frame.py
@@ -536,9 +536,14 @@ class DataFrame(Frame, Generic[T]):
             if index is None:
                 internal = data._internal
         elif isinstance(data, ps.Series):
-            assert columns is None
             assert dtype is None
             assert not copy
+            # For pandas compatibility when `columns` contains only one valid column.
+            if columns is not None:
+                assert isinstance(columns, (dict, list, tuple))
+                assert len(columns) == 1
+                columns = list(columns.keys()) if isinstance(columns, dict) else columns
+                assert columns[0] == data._internal.data_spark_column_names[0]
             if index is None:
                 internal = data.to_frame()._internal
         else:
diff --git a/python/pyspark/pandas/tests/test_dataframe.py b/python/pyspark/pandas/tests/test_dataframe.py
index 1b06d321e13..d33c6584f7f 100644
--- a/python/pyspark/pandas/tests/test_dataframe.py
+++ b/python/pyspark/pandas/tests/test_dataframe.py
@@ -90,6 +90,13 @@ class DataFrameTest(ComparisonTestBase, SQLTestUtils):
         psser = ps.from_pandas(pser)
         self.assert_eq(pd.DataFrame(pser), ps.DataFrame(psser))
 
+        # check ps.DataFrame(ps.Series) with `columns`
+        self.assert_eq(pd.DataFrame(pser, columns=["x"]), ps.DataFrame(psser, columns=["x"]))
+        self.assert_eq(pd.DataFrame(pser, columns=("x",)), ps.DataFrame(psser, columns=("x",)))
+        self.assert_eq(
+            pd.DataFrame(pser, columns={"x": None}), ps.DataFrame(psser, columns={"x": None})
+        )
+
         # check psdf[pd.Index]
         pdf, psdf = self.df_pair
         column_mask = pdf.columns.isin(["a", "b"])


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org