You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/07/26 19:58:23 UTC

[GitHub] [spark] EnricoMi opened a new pull request, #37304: [SPARK-39877][PySpark][WIP] Add unpivot to PySpark DataFrame API

EnricoMi opened a new pull request, #37304:
URL: https://github.com/apache/spark/pull/37304

   ### What changes were proposed in this pull request?
   This adds `unpivot` and its alias `melt` to the PySpark API. It calls into Scala `Dataset.unpivot` (#36150). Small difference to Scala method signature is that PySpark method has default values. This is similar to `melt` in Spark Pandas API.
   
   ### Why are the changes needed?
   To support `unpivot` in Python.
   
   ### Does this PR introduce _any_ user-facing change?
   Yes, adds `DataFrame.unpivot` and `DataFrame.melt` to PySpark API.
   
   ### How was this patch tested?
   Added test to `test_dataframe.py`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on a diff in pull request #37304: [SPARK-39877][PySpark] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on code in PR #37304:
URL: https://github.com/apache/spark/pull/37304#discussion_r930757871


##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -2127,6 +2127,15 @@ class Dataset[T] private[sql](
       valueColumnName: String): DataFrame =
     unpivot(ids, Array.empty, variableColumnName, valueColumnName)
 
+  /**
+   * Called from Python as Seq[Column] are easier to create via py4j than Array[Column].
+   */
+  private[sql] def _unpivot(ids: Seq[Column],

Review Comment:
   maybe we can directly change the parameter types in `unpivot` and `melt` from `Array[Column]` to `Seq[Column]`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a diff in pull request #37304: [SPARK-39877][PYTHON] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #37304:
URL: https://github.com/apache/spark/pull/37304#discussion_r934080682


##########
python/pyspark/sql/dataframe.py:
##########
@@ -2238,6 +2238,140 @@ def cube(self, *cols: "ColumnOrName") -> "GroupedData":  # type: ignore[misc]
 
         return GroupedData(jgd, self)
 
+    def unpivot(

Review Comment:
   Let'd document it under https://github.com/apache/spark/blob/master/python/docs/source/reference/pyspark.sql/dataframe.rst



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on a diff in pull request #37304: [SPARK-39877][PySpark] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on code in PR #37304:
URL: https://github.com/apache/spark/pull/37304#discussion_r933057670


##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -2107,6 +2107,16 @@ class Dataset[T] private[sql](
       valueColumnName: String): DataFrame =
     unpivot(ids, Array.empty, variableColumnName, valueColumnName)
 
+  /**
+   * Called from Python as Seq[Column] are easier to create via py4j than Array[Column].
+   * We use Array[Column] for unpivot rather than Seq[Column] as those are Java-friendly.
+   */
+  private[sql] def unpivotWithSeq(ids: Seq[Column],
+                                  values: Seq[Column],
+                                  variableColumnName: String,
+                                  valueColumnName: String): DataFrame =

Review Comment:
   ```suggestion
     private[sql] def unpivotWithSeq(
         ids: Seq[Column],
         values: Seq[Column],
         variableColumnName: String,
         valueColumnName: String): DataFrame =
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] EnricoMi commented on a diff in pull request #37304: [SPARK-39877][PySpark] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
EnricoMi commented on code in PR #37304:
URL: https://github.com/apache/spark/pull/37304#discussion_r932976862


##########
python/pyspark/context.py:
##########
@@ -309,10 +309,7 @@ def _do_init(
         if sys.version_info[:2] < (3, 8):
             with warnings.catch_warnings():
                 warnings.simplefilter("once")
-                warnings.warn(
-                    "Python 3.7 support is deprecated in Spark 3.4.",
-                    FutureWarning
-                )
+                warnings.warn("Python 3.7 support is deprecated in Spark 3.4.", FutureWarning)

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on a diff in pull request #37304: [SPARK-39877][PySpark][WIP] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on code in PR #37304:
URL: https://github.com/apache/spark/pull/37304#discussion_r930517804


##########
python/pyspark/sql/dataframe.py:
##########
@@ -2188,6 +2188,150 @@ def cube(self, *cols: "ColumnOrName") -> "GroupedData":  # type: ignore[misc]
 
         return GroupedData(jgd, self)
 
+    def unpivot(
+        self,
+        ids: Optional[Union["ColumnOrName", List["ColumnOrName"], Tuple["ColumnOrName"]]] = None,
+        values: Optional[Union["ColumnOrName", List["ColumnOrName"], Tuple["ColumnOrName"]]] = None,
+        variableColumnName: Optional[str] = None,
+        valueColumnName: Optional[str] = None,
+    ) -> "DataFrame":
+        """
+        Unpivot a DataFrame from wide format to long format, optionally leaving
+        identifier columns set. This is the reverse to `groupBy(...).pivot(...).agg(...)`,
+        except for the aggregation, which cannot be reversed.
+
+        This function is useful to massage a DataFrame into a format where some
+        columns are identifier columns ("ids"), while all other columns ("values")
+        are "unpivoted" to the rows, leaving just two non-id columns, named as given
+        by `variableColumnName` and `valueColumnName`.
+
+        When no "id" columns are given, the unpivoted DataFrame consists of only the
+        "variable" and "value" columns.
+
+        All "value" columns must share a least common data type. Unless they are the same data type,
+        all "value" columns are cast to the nearest common data type. For instance, types
+        `IntegerType` and `LongType` are cast to `LongType`, while `IntegerType` and `StringType`
+        do not have a common data type and `unpivot` fails.
+
+        :func:`groupby` is an alias for :func:`groupBy`.
+
+        .. versionadded:: 3.4.0
+
+        Parameters
+        ----------
+        ids : str, Column, tuple, list, optional
+            Column(s) to use as identifiers. Can be a single column or column name,
+            or a list for multiple columns.
+        values : str, Column, tuple, list, optional
+            Column(s) to unpivot. Can be a single column or column name, or a list
+            for multiple columns. If not specified or empty, uses all columns that
+            are not set as `ids`.
+        variableColumnName : scalar, default 'variable'
+            Name of the variable column. If None it uses 'variable'.
+        valueColumnName : scalar, default 'value'
+            Name of the value column. If None it uses 'value'.
+
+        Returns
+        -------
+        DataFrame
+            Unpivoted DataFrame.
+
+        Examples
+        --------
+        >>> df = spark.createDataFrame(
+        ...     [(1, 11, 1.1), (2, 12, 1.2)],
+        ...     ["id", "int", "double"],
+        ... )
+        >>> df.show()
+        +---+---+------+
+        | id|int|double|
+        +---+---+------+
+        |  1| 11|   1.1|
+        |  2| 12|   1.2|
+        +---+---+------+
+
+        >>> df.unpivot("id").show()
+        +---+--------+-----+
+        | id|variable|value|
+        +---+--------+-----+
+        |  1|     int| 11.0|
+        |  1|  double|  1.1|
+        |  2|     int| 12.0|
+        |  2|  double|  1.2|
+        +---+--------+-----+
+        """
+        if ids is None:
+            id_list = []
+        elif isinstance(ids, tuple):
+            id_list = list(ids)
+        elif isinstance(ids, list):
+            id_list = ids
+        else:
+            id_list = [ids]
+
+        if values is None:
+            value_list = []
+        elif isinstance(values, tuple):
+            value_list = list(values)
+        elif isinstance(values, list):
+            value_list = values
+        else:
+            value_list = [values]
+
+        if variableColumnName is None:
+            variableColumnName = "variable"
+
+        if valueColumnName is None:
+            valueColumnName = "value"
+
+        return DataFrame(
+            self._jdf._unpivot(
+                self._jcols(*id_list), self._jcols(*value_list), variableColumnName, valueColumnName
+            ),
+            self.sparkSession,
+        )
+
+    def melt(

Review Comment:
   there was also a `melt` function in `python/pyspark/pandas/frame.py`, is it possbile to apply the new `unpivot` implementation there?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] EnricoMi commented on a diff in pull request #37304: [SPARK-39877][PySpark][WIP] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
EnricoMi commented on code in PR #37304:
URL: https://github.com/apache/spark/pull/37304#discussion_r930677635


##########
python/pyspark/sql/dataframe.py:
##########
@@ -2188,6 +2188,150 @@ def cube(self, *cols: "ColumnOrName") -> "GroupedData":  # type: ignore[misc]
 
         return GroupedData(jgd, self)
 
+    def unpivot(
+        self,
+        ids: Optional[Union["ColumnOrName", List["ColumnOrName"], Tuple["ColumnOrName"]]] = None,
+        values: Optional[Union["ColumnOrName", List["ColumnOrName"], Tuple["ColumnOrName"]]] = None,
+        variableColumnName: Optional[str] = None,
+        valueColumnName: Optional[str] = None,
+    ) -> "DataFrame":
+        """
+        Unpivot a DataFrame from wide format to long format, optionally leaving
+        identifier columns set. This is the reverse to `groupBy(...).pivot(...).agg(...)`,
+        except for the aggregation, which cannot be reversed.
+
+        This function is useful to massage a DataFrame into a format where some
+        columns are identifier columns ("ids"), while all other columns ("values")
+        are "unpivoted" to the rows, leaving just two non-id columns, named as given
+        by `variableColumnName` and `valueColumnName`.
+
+        When no "id" columns are given, the unpivoted DataFrame consists of only the
+        "variable" and "value" columns.
+
+        All "value" columns must share a least common data type. Unless they are the same data type,
+        all "value" columns are cast to the nearest common data type. For instance, types
+        `IntegerType` and `LongType` are cast to `LongType`, while `IntegerType` and `StringType`
+        do not have a common data type and `unpivot` fails.
+
+        :func:`groupby` is an alias for :func:`groupBy`.
+
+        .. versionadded:: 3.4.0
+
+        Parameters
+        ----------
+        ids : str, Column, tuple, list, optional
+            Column(s) to use as identifiers. Can be a single column or column name,
+            or a list for multiple columns.
+        values : str, Column, tuple, list, optional
+            Column(s) to unpivot. Can be a single column or column name, or a list
+            for multiple columns. If not specified or empty, uses all columns that
+            are not set as `ids`.
+        variableColumnName : scalar, default 'variable'
+            Name of the variable column. If None it uses 'variable'.
+        valueColumnName : scalar, default 'value'
+            Name of the value column. If None it uses 'value'.
+
+        Returns
+        -------
+        DataFrame
+            Unpivoted DataFrame.
+
+        Examples
+        --------
+        >>> df = spark.createDataFrame(
+        ...     [(1, 11, 1.1), (2, 12, 1.2)],
+        ...     ["id", "int", "double"],
+        ... )
+        >>> df.show()
+        +---+---+------+
+        | id|int|double|
+        +---+---+------+
+        |  1| 11|   1.1|
+        |  2| 12|   1.2|
+        +---+---+------+
+
+        >>> df.unpivot("id").show()
+        +---+--------+-----+
+        | id|variable|value|
+        +---+--------+-----+
+        |  1|     int| 11.0|
+        |  1|  double|  1.1|
+        |  2|     int| 12.0|
+        |  2|  double|  1.2|
+        +---+--------+-----+
+        """
+        if ids is None:
+            id_list = []
+        elif isinstance(ids, tuple):
+            id_list = list(ids)
+        elif isinstance(ids, list):
+            id_list = ids
+        else:
+            id_list = [ids]
+
+        if values is None:
+            value_list = []
+        elif isinstance(values, tuple):
+            value_list = list(values)
+        elif isinstance(values, list):
+            value_list = values
+        else:
+            value_list = [values]
+
+        if variableColumnName is None:
+            variableColumnName = "variable"
+
+        if valueColumnName is None:
+            valueColumnName = "value"
+
+        return DataFrame(
+            self._jdf._unpivot(
+                self._jcols(*id_list), self._jcols(*value_list), variableColumnName, valueColumnName
+            ),
+            self.sparkSession,
+        )
+
+    def melt(

Review Comment:
   That is planned, but I'd prefer to do that in a separate PR: SPARK-39878



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on a diff in pull request #37304: [SPARK-39877][PySpark] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on code in PR #37304:
URL: https://github.com/apache/spark/pull/37304#discussion_r931757563


##########
python/pyspark/sql/dataframe.py:
##########
@@ -2188,6 +2188,142 @@ def cube(self, *cols: "ColumnOrName") -> "GroupedData":  # type: ignore[misc]
 
         return GroupedData(jgd, self)
 
+    def unpivot(
+        self,
+        ids: Optional[Union["ColumnOrName", List["ColumnOrName"], Tuple["ColumnOrName", ...]]] = None,
+        values: Optional[Union["ColumnOrName", List["ColumnOrName"], Tuple["ColumnOrName", ...]]] = None,

Review Comment:
   I'm netural on it.
   
   some existing methods also use tuple of column names:
   
   ```
       def freqItems(
           self, cols: Union[List[str], Tuple[str]], support: Optional[float] = None
       ) -> "DataFrame":
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on a diff in pull request #37304: [SPARK-39877][PySpark] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on code in PR #37304:
URL: https://github.com/apache/spark/pull/37304#discussion_r931759379


##########
python/pyspark/sql/dataframe.py:
##########
@@ -2188,6 +2188,142 @@ def cube(self, *cols: "ColumnOrName") -> "GroupedData":  # type: ignore[misc]
 
         return GroupedData(jgd, self)
 
+    def unpivot(
+        self,
+        ids: Optional[Union["ColumnOrName", List["ColumnOrName"], Tuple["ColumnOrName", ...]]] = None,
+        values: Optional[Union["ColumnOrName", List["ColumnOrName"], Tuple["ColumnOrName", ...]]] = None,
+        variableColumnName: Optional[str] = None,
+        valueColumnName: Optional[str] = None,
+    ) -> "DataFrame":
+        """
+        Unpivot a DataFrame from wide format to long format, optionally leaving
+        identifier columns set. This is the reverse to `groupBy(...).pivot(...).agg(...)`,
+        except for the aggregation, which cannot be reversed.
+
+        This function is useful to massage a DataFrame into a format where some
+        columns are identifier columns ("ids"), while all other columns ("values")
+        are "unpivoted" to the rows, leaving just two non-id columns, named as given
+        by `variableColumnName` and `valueColumnName`.
+
+        When no "id" columns are given, the unpivoted DataFrame consists of only the
+        "variable" and "value" columns.
+
+        All "value" columns must share a least common data type. Unless they are the same data type,
+        all "value" columns are cast to the nearest common data type. For instance, types
+        `IntegerType` and `LongType` are cast to `LongType`, while `IntegerType` and `StringType`
+        do not have a common data type and `unpivot` fails.
+
+        :func:`groupby` is an alias for :func:`groupBy`.
+
+        .. versionadded:: 3.4.0
+
+        Parameters
+        ----------
+        ids : str, Column, tuple, list, optional
+            Column(s) to use as identifiers. Can be a single column or column name,
+            or a list for multiple columns.
+        values : str, Column, tuple, list, optional
+            Column(s) to unpivot. Can be a single column or column name, or a list
+            for multiple columns. If not specified or empty, uses all columns that
+            are not set as `ids`.
+        variableColumnName : scalar, default 'variable'
+            Name of the variable column. If None it uses 'variable'.
+        valueColumnName : scalar, default 'value'
+            Name of the value column. If None it uses 'value'.
+
+        Returns
+        -------
+        DataFrame
+            Unpivoted DataFrame.
+
+        Examples
+        --------
+        >>> df = spark.createDataFrame(
+        ...     [(1, 11, 1.1), (2, 12, 1.2)],
+        ...     ["id", "int", "double"],
+        ... )
+        >>> df.show()
+        +---+---+------+
+        | id|int|double|
+        +---+---+------+
+        |  1| 11|   1.1|
+        |  2| 12|   1.2|
+        +---+---+------+
+
+        >>> df.unpivot("id").show()
+        +---+--------+-----+
+        | id|variable|value|
+        +---+--------+-----+
+        |  1|     int| 11.0|
+        |  1|  double|  1.1|
+        |  2|     int| 12.0|
+        |  2|  double|  1.2|
+        +---+--------+-----+
+        """
+        def to_jcols(cols) -> JavaObject:
+            l = cols
+            if cols is None:
+                l = []
+            elif isinstance(cols, tuple):
+                l = list(cols)
+            elif not isinstance(cols, list):
+                l = [cols]
+            return self._jcols(*l)
+
+        if variableColumnName is None:
+            variableColumnName = "variable"

Review Comment:
   i think we'd better keep the behaviors the same. I mean, PySpark Dataframe API to match Scala Dataset API



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] EnricoMi commented on a diff in pull request #37304: [SPARK-39877][PYTHON] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
EnricoMi commented on code in PR #37304:
URL: https://github.com/apache/spark/pull/37304#discussion_r934227128


##########
python/pyspark/sql/dataframe.py:
##########
@@ -2238,6 +2238,140 @@ def cube(self, *cols: "ColumnOrName") -> "GroupedData":  # type: ignore[misc]
 
         return GroupedData(jgd, self)
 
+    def unpivot(

Review Comment:
   https://github.com/apache/spark/pull/37354



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on a diff in pull request #37304: [SPARK-39877][PySpark] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on code in PR #37304:
URL: https://github.com/apache/spark/pull/37304#discussion_r932846004


##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -2127,6 +2127,15 @@ class Dataset[T] private[sql](
       valueColumnName: String): DataFrame =
     unpivot(ids, Array.empty, variableColumnName, valueColumnName)
 
+  /**
+   * Called from Python as Seq[Column] are easier to create via py4j than Array[Column].
+   */
+  private[sql] def _unpivot(ids: Seq[Column],

Review Comment:
   can you rename the method? we do not use `_xxx` like function names in scala
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] EnricoMi commented on a diff in pull request #37304: [SPARK-39877][PySpark] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
EnricoMi commented on code in PR #37304:
URL: https://github.com/apache/spark/pull/37304#discussion_r932044028


##########
python/pyspark/sql/dataframe.py:
##########
@@ -2188,6 +2188,142 @@ def cube(self, *cols: "ColumnOrName") -> "GroupedData":  # type: ignore[misc]
 
         return GroupedData(jgd, self)
 
+    def unpivot(
+        self,
+        ids: Optional[Union["ColumnOrName", List["ColumnOrName"], Tuple["ColumnOrName", ...]]] = None,
+        values: Optional[Union["ColumnOrName", List["ColumnOrName"], Tuple["ColumnOrName", ...]]] = None,
+        variableColumnName: Optional[str] = None,
+        valueColumnName: Optional[str] = None,
+    ) -> "DataFrame":
+        """
+        Unpivot a DataFrame from wide format to long format, optionally leaving
+        identifier columns set. This is the reverse to `groupBy(...).pivot(...).agg(...)`,
+        except for the aggregation, which cannot be reversed.
+
+        This function is useful to massage a DataFrame into a format where some
+        columns are identifier columns ("ids"), while all other columns ("values")
+        are "unpivoted" to the rows, leaving just two non-id columns, named as given
+        by `variableColumnName` and `valueColumnName`.
+
+        When no "id" columns are given, the unpivoted DataFrame consists of only the
+        "variable" and "value" columns.
+
+        All "value" columns must share a least common data type. Unless they are the same data type,
+        all "value" columns are cast to the nearest common data type. For instance, types
+        `IntegerType` and `LongType` are cast to `LongType`, while `IntegerType` and `StringType`
+        do not have a common data type and `unpivot` fails.
+
+        :func:`groupby` is an alias for :func:`groupBy`.
+
+        .. versionadded:: 3.4.0
+
+        Parameters
+        ----------
+        ids : str, Column, tuple, list, optional
+            Column(s) to use as identifiers. Can be a single column or column name,
+            or a list for multiple columns.
+        values : str, Column, tuple, list, optional
+            Column(s) to unpivot. Can be a single column or column name, or a list
+            for multiple columns. If not specified or empty, uses all columns that
+            are not set as `ids`.
+        variableColumnName : scalar, default 'variable'
+            Name of the variable column. If None it uses 'variable'.
+        valueColumnName : scalar, default 'value'
+            Name of the value column. If None it uses 'value'.
+
+        Returns
+        -------
+        DataFrame
+            Unpivoted DataFrame.
+
+        Examples
+        --------
+        >>> df = spark.createDataFrame(
+        ...     [(1, 11, 1.1), (2, 12, 1.2)],
+        ...     ["id", "int", "double"],
+        ... )
+        >>> df.show()
+        +---+---+------+
+        | id|int|double|
+        +---+---+------+
+        |  1| 11|   1.1|
+        |  2| 12|   1.2|
+        +---+---+------+
+
+        >>> df.unpivot("id").show()
+        +---+--------+-----+
+        | id|variable|value|
+        +---+--------+-----+
+        |  1|     int| 11.0|
+        |  1|  double|  1.1|
+        |  2|     int| 12.0|
+        |  2|  double|  1.2|
+        +---+--------+-----+
+        """
+        def to_jcols(cols) -> JavaObject:
+            l = cols
+            if cols is None:
+                l = []
+            elif isinstance(cols, tuple):
+                l = list(cols)
+            elif not isinstance(cols, list):
+                l = [cols]
+            return self._jcols(*l)
+
+        if variableColumnName is None:
+            variableColumnName = "variable"

Review Comment:
   removed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on a diff in pull request #37304: [SPARK-39877][PySpark] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on code in PR #37304:
URL: https://github.com/apache/spark/pull/37304#discussion_r930757871


##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -2127,6 +2127,15 @@ class Dataset[T] private[sql](
       valueColumnName: String): DataFrame =
     unpivot(ids, Array.empty, variableColumnName, valueColumnName)
 
+  /**
+   * Called from Python as Seq[Column] are easier to create via py4j than Array[Column].
+   */
+  private[sql] def _unpivot(ids: Seq[Column],

Review Comment:
   ~~maybe we can directly change the parameter types in `unpivot` and `melt` from `Array[Column]` to `Seq[Column]`?~~



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] itholic commented on a diff in pull request #37304: [SPARK-39877][PySpark] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
itholic commented on code in PR #37304:
URL: https://github.com/apache/spark/pull/37304#discussion_r930842298


##########
python/pyspark/sql/dataframe.py:
##########
@@ -2188,6 +2188,142 @@ def cube(self, *cols: "ColumnOrName") -> "GroupedData":  # type: ignore[misc]
 
         return GroupedData(jgd, self)
 
+    def unpivot(
+        self,
+        ids: Optional[Union["ColumnOrName", List["ColumnOrName"], Tuple["ColumnOrName", ...]]] = None,
+        values: Optional[Union["ColumnOrName", List["ColumnOrName"], Tuple["ColumnOrName", ...]]] = None,
+        variableColumnName: Optional[str] = None,
+        valueColumnName: Optional[str] = None,
+    ) -> "DataFrame":
+        """
+        Unpivot a DataFrame from wide format to long format, optionally leaving
+        identifier columns set. This is the reverse to `groupBy(...).pivot(...).agg(...)`,
+        except for the aggregation, which cannot be reversed.
+
+        This function is useful to massage a DataFrame into a format where some
+        columns are identifier columns ("ids"), while all other columns ("values")
+        are "unpivoted" to the rows, leaving just two non-id columns, named as given
+        by `variableColumnName` and `valueColumnName`.
+
+        When no "id" columns are given, the unpivoted DataFrame consists of only the
+        "variable" and "value" columns.
+
+        All "value" columns must share a least common data type. Unless they are the same data type,
+        all "value" columns are cast to the nearest common data type. For instance, types
+        `IntegerType` and `LongType` are cast to `LongType`, while `IntegerType` and `StringType`
+        do not have a common data type and `unpivot` fails.
+
+        :func:`groupby` is an alias for :func:`groupBy`.
+
+        .. versionadded:: 3.4.0
+
+        Parameters
+        ----------
+        ids : str, Column, tuple, list, optional
+            Column(s) to use as identifiers. Can be a single column or column name,
+            or a list for multiple columns.
+        values : str, Column, tuple, list, optional
+            Column(s) to unpivot. Can be a single column or column name, or a list
+            for multiple columns. If not specified or empty, uses all columns that
+            are not set as `ids`.
+        variableColumnName : scalar, default 'variable'
+            Name of the variable column. If None it uses 'variable'.
+        valueColumnName : scalar, default 'value'
+            Name of the value column. If None it uses 'value'.
+
+        Returns
+        -------
+        DataFrame
+            Unpivoted DataFrame.
+
+        Examples
+        --------
+        >>> df = spark.createDataFrame(
+        ...     [(1, 11, 1.1), (2, 12, 1.2)],
+        ...     ["id", "int", "double"],
+        ... )
+        >>> df.show()
+        +---+---+------+
+        | id|int|double|
+        +---+---+------+
+        |  1| 11|   1.1|
+        |  2| 12|   1.2|
+        +---+---+------+
+
+        >>> df.unpivot("id").show()
+        +---+--------+-----+
+        | id|variable|value|
+        +---+--------+-----+
+        |  1|     int| 11.0|
+        |  1|  double|  1.1|
+        |  2|     int| 12.0|
+        |  2|  double|  1.2|
+        +---+--------+-----+
+        """
+        def to_jcols(cols) -> JavaObject:
+            l = cols
+            if cols is None:
+                l = []
+            elif isinstance(cols, tuple):
+                l = list(cols)
+            elif not isinstance(cols, list):
+                l = [cols]
+            return self._jcols(*l)
+
+        if variableColumnName is None:
+            variableColumnName = "variable"

Review Comment:
   > Scala side doesn't have similar logic?
   
   Seems like Scala side doesn't have similar logic according to the [implementation](https://github.com/apache/spark/pull/36150/files) ?
   
   Maybe we should match the default value?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #37304: [SPARK-39877][PYTHON] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #37304:
URL: https://github.com/apache/spark/pull/37304#issuecomment-1200590520

   LGTM2


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] EnricoMi commented on a diff in pull request #37304: [SPARK-39877][PySpark][WIP] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
EnricoMi commented on code in PR #37304:
URL: https://github.com/apache/spark/pull/37304#discussion_r930678850


##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -2127,6 +2127,15 @@ class Dataset[T] private[sql](
       valueColumnName: String): DataFrame =
     unpivot(ids, Array.empty, variableColumnName, valueColumnName)
 
+  /**
+   * Called from Python as Seq[Column] are easier to create via py4j than Array[Column].
+   */
+  private[sql] def _unpivot(ids: Seq[Column],

Review Comment:
   I'd prefer to get `PythonSQLUtils.toArray` return array of `Column`s but it returns array of `Object`. Looks like generic types do not work as expected.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] EnricoMi commented on a diff in pull request #37304: [SPARK-39877][PySpark] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
EnricoMi commented on code in PR #37304:
URL: https://github.com/apache/spark/pull/37304#discussion_r931873169


##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -2127,6 +2127,15 @@ class Dataset[T] private[sql](
       valueColumnName: String): DataFrame =
     unpivot(ids, Array.empty, variableColumnName, valueColumnName)
 
+  /**
+   * Called from Python as Seq[Column] are easier to create via py4j than Array[Column].
+   */
+  private[sql] def _unpivot(ids: Seq[Column],

Review Comment:
   Sure, just hesitating because that looks like a very specific method for such a generic utils class. The other methods are more generic. And there are other private Python methods in `Dataset` (`mapInPandas`) and `RelationalGroupedDataset` (`flatMapGroupsInPandas`).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng closed pull request #37304: [SPARK-39877][PYTHON] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
zhengruifeng closed pull request #37304: [SPARK-39877][PYTHON] Add unpivot to PySpark DataFrame API
URL: https://github.com/apache/spark/pull/37304


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on a diff in pull request #37304: [SPARK-39877][PySpark] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on code in PR #37304:
URL: https://github.com/apache/spark/pull/37304#discussion_r931760989


##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -2127,6 +2127,15 @@ class Dataset[T] private[sql](
       valueColumnName: String): DataFrame =
     unpivot(ids, Array.empty, variableColumnName, valueColumnName)
 
+  /**
+   * Called from Python as Seq[Column] are easier to create via py4j than Array[Column].
+   */
+  private[sql] def _unpivot(ids: Seq[Column],

Review Comment:
   got it.
   
   since this method is still needed and dedicated only for python, what about move it to `PythonSQLUtils`?
   
   something like
   
   ```
   def unpivot(dataframe: DataFrame, ids: Seq[Column] ...) = {
       dataframe.unpivot(ids.toArray, values.toArray, variableColumnName, valueColumnName)
   }
   
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on a diff in pull request #37304: [SPARK-39877][PySpark] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on code in PR #37304:
URL: https://github.com/apache/spark/pull/37304#discussion_r931759379


##########
python/pyspark/sql/dataframe.py:
##########
@@ -2188,6 +2188,142 @@ def cube(self, *cols: "ColumnOrName") -> "GroupedData":  # type: ignore[misc]
 
         return GroupedData(jgd, self)
 
+    def unpivot(
+        self,
+        ids: Optional[Union["ColumnOrName", List["ColumnOrName"], Tuple["ColumnOrName", ...]]] = None,
+        values: Optional[Union["ColumnOrName", List["ColumnOrName"], Tuple["ColumnOrName", ...]]] = None,
+        variableColumnName: Optional[str] = None,
+        valueColumnName: Optional[str] = None,
+    ) -> "DataFrame":
+        """
+        Unpivot a DataFrame from wide format to long format, optionally leaving
+        identifier columns set. This is the reverse to `groupBy(...).pivot(...).agg(...)`,
+        except for the aggregation, which cannot be reversed.
+
+        This function is useful to massage a DataFrame into a format where some
+        columns are identifier columns ("ids"), while all other columns ("values")
+        are "unpivoted" to the rows, leaving just two non-id columns, named as given
+        by `variableColumnName` and `valueColumnName`.
+
+        When no "id" columns are given, the unpivoted DataFrame consists of only the
+        "variable" and "value" columns.
+
+        All "value" columns must share a least common data type. Unless they are the same data type,
+        all "value" columns are cast to the nearest common data type. For instance, types
+        `IntegerType` and `LongType` are cast to `LongType`, while `IntegerType` and `StringType`
+        do not have a common data type and `unpivot` fails.
+
+        :func:`groupby` is an alias for :func:`groupBy`.
+
+        .. versionadded:: 3.4.0
+
+        Parameters
+        ----------
+        ids : str, Column, tuple, list, optional
+            Column(s) to use as identifiers. Can be a single column or column name,
+            or a list for multiple columns.
+        values : str, Column, tuple, list, optional
+            Column(s) to unpivot. Can be a single column or column name, or a list
+            for multiple columns. If not specified or empty, uses all columns that
+            are not set as `ids`.
+        variableColumnName : scalar, default 'variable'
+            Name of the variable column. If None it uses 'variable'.
+        valueColumnName : scalar, default 'value'
+            Name of the value column. If None it uses 'value'.
+
+        Returns
+        -------
+        DataFrame
+            Unpivoted DataFrame.
+
+        Examples
+        --------
+        >>> df = spark.createDataFrame(
+        ...     [(1, 11, 1.1), (2, 12, 1.2)],
+        ...     ["id", "int", "double"],
+        ... )
+        >>> df.show()
+        +---+---+------+
+        | id|int|double|
+        +---+---+------+
+        |  1| 11|   1.1|
+        |  2| 12|   1.2|
+        +---+---+------+
+
+        >>> df.unpivot("id").show()
+        +---+--------+-----+
+        | id|variable|value|
+        +---+--------+-----+
+        |  1|     int| 11.0|
+        |  1|  double|  1.1|
+        |  2|     int| 12.0|
+        |  2|  double|  1.2|
+        +---+--------+-----+
+        """
+        def to_jcols(cols) -> JavaObject:
+            l = cols
+            if cols is None:
+                l = []
+            elif isinstance(cols, tuple):
+                l = list(cols)
+            elif not isinstance(cols, list):
+                l = [cols]
+            return self._jcols(*l)
+
+        if variableColumnName is None:
+            variableColumnName = "variable"

Review Comment:
   i think we'd better keep the behaviors the same.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on a diff in pull request #37304: [SPARK-39877][PySpark] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on code in PR #37304:
URL: https://github.com/apache/spark/pull/37304#discussion_r930800109


##########
python/pyspark/sql/dataframe.py:
##########
@@ -2188,6 +2188,142 @@ def cube(self, *cols: "ColumnOrName") -> "GroupedData":  # type: ignore[misc]
 
         return GroupedData(jgd, self)
 
+    def unpivot(
+        self,
+        ids: Optional[Union["ColumnOrName", List["ColumnOrName"], Tuple["ColumnOrName", ...]]] = None,
+        values: Optional[Union["ColumnOrName", List["ColumnOrName"], Tuple["ColumnOrName", ...]]] = None,
+        variableColumnName: Optional[str] = None,
+        valueColumnName: Optional[str] = None,
+    ) -> "DataFrame":
+        """
+        Unpivot a DataFrame from wide format to long format, optionally leaving
+        identifier columns set. This is the reverse to `groupBy(...).pivot(...).agg(...)`,
+        except for the aggregation, which cannot be reversed.
+
+        This function is useful to massage a DataFrame into a format where some
+        columns are identifier columns ("ids"), while all other columns ("values")
+        are "unpivoted" to the rows, leaving just two non-id columns, named as given
+        by `variableColumnName` and `valueColumnName`.
+
+        When no "id" columns are given, the unpivoted DataFrame consists of only the
+        "variable" and "value" columns.
+
+        All "value" columns must share a least common data type. Unless they are the same data type,
+        all "value" columns are cast to the nearest common data type. For instance, types
+        `IntegerType` and `LongType` are cast to `LongType`, while `IntegerType` and `StringType`
+        do not have a common data type and `unpivot` fails.
+
+        :func:`groupby` is an alias for :func:`groupBy`.
+
+        .. versionadded:: 3.4.0
+
+        Parameters
+        ----------
+        ids : str, Column, tuple, list, optional
+            Column(s) to use as identifiers. Can be a single column or column name,
+            or a list for multiple columns.
+        values : str, Column, tuple, list, optional
+            Column(s) to unpivot. Can be a single column or column name, or a list
+            for multiple columns. If not specified or empty, uses all columns that
+            are not set as `ids`.
+        variableColumnName : scalar, default 'variable'
+            Name of the variable column. If None it uses 'variable'.
+        valueColumnName : scalar, default 'value'
+            Name of the value column. If None it uses 'value'.
+
+        Returns
+        -------
+        DataFrame
+            Unpivoted DataFrame.
+
+        Examples
+        --------
+        >>> df = spark.createDataFrame(
+        ...     [(1, 11, 1.1), (2, 12, 1.2)],
+        ...     ["id", "int", "double"],
+        ... )
+        >>> df.show()
+        +---+---+------+
+        | id|int|double|
+        +---+---+------+
+        |  1| 11|   1.1|
+        |  2| 12|   1.2|
+        +---+---+------+
+
+        >>> df.unpivot("id").show()
+        +---+--------+-----+
+        | id|variable|value|
+        +---+--------+-----+
+        |  1|     int| 11.0|
+        |  1|  double|  1.1|
+        |  2|     int| 12.0|
+        |  2|  double|  1.2|
+        +---+--------+-----+
+        """
+        def to_jcols(cols) -> JavaObject:
+            l = cols
+            if cols is None:
+                l = []
+            elif isinstance(cols, tuple):
+                l = list(cols)
+            elif not isinstance(cols, list):
+                l = [cols]
+            return self._jcols(*l)
+
+        if variableColumnName is None:
+            variableColumnName = "variable"

Review Comment:
   default values `variableColumnName = "variable"` and `valueColumnName = "value"` are only used in pyspark?
   
   Scala side doesn't have similar logic?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] EnricoMi commented on a diff in pull request #37304: [SPARK-39877][PySpark] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
EnricoMi commented on code in PR #37304:
URL: https://github.com/apache/spark/pull/37304#discussion_r930979982


##########
python/pyspark/sql/dataframe.py:
##########
@@ -2188,6 +2188,142 @@ def cube(self, *cols: "ColumnOrName") -> "GroupedData":  # type: ignore[misc]
 
         return GroupedData(jgd, self)
 
+    def unpivot(
+        self,
+        ids: Optional[Union["ColumnOrName", List["ColumnOrName"], Tuple["ColumnOrName", ...]]] = None,
+        values: Optional[Union["ColumnOrName", List["ColumnOrName"], Tuple["ColumnOrName", ...]]] = None,
+        variableColumnName: Optional[str] = None,
+        valueColumnName: Optional[str] = None,
+    ) -> "DataFrame":
+        """
+        Unpivot a DataFrame from wide format to long format, optionally leaving
+        identifier columns set. This is the reverse to `groupBy(...).pivot(...).agg(...)`,
+        except for the aggregation, which cannot be reversed.
+
+        This function is useful to massage a DataFrame into a format where some
+        columns are identifier columns ("ids"), while all other columns ("values")
+        are "unpivoted" to the rows, leaving just two non-id columns, named as given
+        by `variableColumnName` and `valueColumnName`.
+
+        When no "id" columns are given, the unpivoted DataFrame consists of only the
+        "variable" and "value" columns.
+
+        All "value" columns must share a least common data type. Unless they are the same data type,
+        all "value" columns are cast to the nearest common data type. For instance, types
+        `IntegerType` and `LongType` are cast to `LongType`, while `IntegerType` and `StringType`
+        do not have a common data type and `unpivot` fails.
+
+        :func:`groupby` is an alias for :func:`groupBy`.
+
+        .. versionadded:: 3.4.0
+
+        Parameters
+        ----------
+        ids : str, Column, tuple, list, optional
+            Column(s) to use as identifiers. Can be a single column or column name,
+            or a list for multiple columns.
+        values : str, Column, tuple, list, optional
+            Column(s) to unpivot. Can be a single column or column name, or a list
+            for multiple columns. If not specified or empty, uses all columns that
+            are not set as `ids`.
+        variableColumnName : scalar, default 'variable'
+            Name of the variable column. If None it uses 'variable'.
+        valueColumnName : scalar, default 'value'
+            Name of the value column. If None it uses 'value'.
+
+        Returns
+        -------
+        DataFrame
+            Unpivoted DataFrame.
+
+        Examples
+        --------
+        >>> df = spark.createDataFrame(
+        ...     [(1, 11, 1.1), (2, 12, 1.2)],
+        ...     ["id", "int", "double"],
+        ... )
+        >>> df.show()
+        +---+---+------+
+        | id|int|double|
+        +---+---+------+
+        |  1| 11|   1.1|
+        |  2| 12|   1.2|
+        +---+---+------+
+
+        >>> df.unpivot("id").show()
+        +---+--------+-----+
+        | id|variable|value|
+        +---+--------+-----+
+        |  1|     int| 11.0|
+        |  1|  double|  1.1|
+        |  2|     int| 12.0|
+        |  2|  double|  1.2|
+        +---+--------+-----+
+        """
+        def to_jcols(cols) -> JavaObject:
+            l = cols
+            if cols is None:
+                l = []
+            elif isinstance(cols, tuple):
+                l = list(cols)
+            elif not isinstance(cols, list):
+                l = [cols]
+            return self._jcols(*l)
+
+        if variableColumnName is None:
+            variableColumnName = "variable"

Review Comment:
   The rational in Scala was that other SQL systems require variable and value column names (there is no default), so we decided Scala API does not have default values either. Given that PySpark Pandas API has default values I thought PySpark Dataframe API could have one too.
   
   If you prefer PySpark Dataframe API to match Scala Dataset API rather than PySpark Pandas API, I am fine with that.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] EnricoMi commented on a diff in pull request #37304: [SPARK-39877][PySpark] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
EnricoMi commented on code in PR #37304:
URL: https://github.com/apache/spark/pull/37304#discussion_r930982032


##########
python/pyspark/sql/dataframe.py:
##########
@@ -2188,6 +2188,142 @@ def cube(self, *cols: "ColumnOrName") -> "GroupedData":  # type: ignore[misc]
 
         return GroupedData(jgd, self)
 
+    def unpivot(
+        self,
+        ids: Optional[Union["ColumnOrName", List["ColumnOrName"], Tuple["ColumnOrName", ...]]] = None,
+        values: Optional[Union["ColumnOrName", List["ColumnOrName"], Tuple["ColumnOrName", ...]]] = None,

Review Comment:
   What are your thoughts on allowing for tuple of columns as an alternative to list of columns (equivalent semantics)? `Dataframe._jcols` does not support tuples, only list and single string so this does not seem to be standard behaviour.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on pull request #37304: [SPARK-39877][PySpark] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on PR #37304:
URL: https://github.com/apache/spark/pull/37304#issuecomment-1197631016

   btw, you may also need to run `dev/reformat-python` 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #37304: [SPARK-39877][PySpark] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on PR #37304:
URL: https://github.com/apache/spark/pull/37304#issuecomment-1196747342

   Can one of the admins verify this patch?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on pull request #37304: [SPARK-39877][PYTHON] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on PR #37304:
URL: https://github.com/apache/spark/pull/37304#issuecomment-1200583537

   Merged to master, thanks @EnricoMi for working on this!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] EnricoMi commented on pull request #37304: [SPARK-39877][PySpark] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
EnricoMi commented on PR #37304:
URL: https://github.com/apache/spark/pull/37304#issuecomment-1198506801

   > btw, you may also need to run `dev/reformat-python`
   
   Why do I have to reformat `python/pyspark/context.py`? That seems unrelated.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on a diff in pull request #37304: [SPARK-39877][PySpark][WIP] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on code in PR #37304:
URL: https://github.com/apache/spark/pull/37304#discussion_r930518305


##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -2127,6 +2127,15 @@ class Dataset[T] private[sql](
       valueColumnName: String): DataFrame =
     unpivot(ids, Array.empty, variableColumnName, valueColumnName)
 
+  /**
+   * Called from Python as Seq[Column] are easier to create via py4j than Array[Column].
+   */
+  private[sql] def _unpivot(ids: Seq[Column],

Review Comment:
   I guess we can move it to `src/main/scala/org/apache/spark/sql/api/python/PythonSQLUtils.scala`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] EnricoMi commented on a diff in pull request #37304: [SPARK-39877][PySpark] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
EnricoMi commented on code in PR #37304:
URL: https://github.com/apache/spark/pull/37304#discussion_r930678850


##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -2127,6 +2127,15 @@ class Dataset[T] private[sql](
       valueColumnName: String): DataFrame =
     unpivot(ids, Array.empty, variableColumnName, valueColumnName)
 
+  /**
+   * Called from Python as Seq[Column] are easier to create via py4j than Array[Column].
+   */
+  private[sql] def _unpivot(ids: Seq[Column],

Review Comment:
   I'd prefer to get `PythonSQLUtils.toArray` return array of `Column`s but it returns array of `Object`. Then, this private method would be not needed anymore. Looks like generic types do not work as expected.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on a diff in pull request #37304: [SPARK-39877][PySpark] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on code in PR #37304:
URL: https://github.com/apache/spark/pull/37304#discussion_r930757871


##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -2127,6 +2127,15 @@ class Dataset[T] private[sql](
       valueColumnName: String): DataFrame =
     unpivot(ids, Array.empty, variableColumnName, valueColumnName)
 
+  /**
+   * Called from Python as Seq[Column] are easier to create via py4j than Array[Column].
+   */
+  private[sql] def _unpivot(ids: Seq[Column],

Review Comment:
   maybe we can directly change the types of `ids` and `values` from `Array[Column]` to `Seq[Column]`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] EnricoMi commented on a diff in pull request #37304: [SPARK-39877][PySpark] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
EnricoMi commented on code in PR #37304:
URL: https://github.com/apache/spark/pull/37304#discussion_r930781960


##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -2127,6 +2127,15 @@ class Dataset[T] private[sql](
       valueColumnName: String): DataFrame =
     unpivot(ids, Array.empty, variableColumnName, valueColumnName)
 
+  /**
+   * Called from Python as Seq[Column] are easier to create via py4j than Array[Column].
+   */
+  private[sql] def _unpivot(ids: Seq[Column],

Review Comment:
   The `Array[Column]` was chosen over `Seq[Column]` to make `Dataset.unpivot` Java-friendly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] itholic commented on a diff in pull request #37304: [SPARK-39877][PySpark] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
itholic commented on code in PR #37304:
URL: https://github.com/apache/spark/pull/37304#discussion_r930842298


##########
python/pyspark/sql/dataframe.py:
##########
@@ -2188,6 +2188,142 @@ def cube(self, *cols: "ColumnOrName") -> "GroupedData":  # type: ignore[misc]
 
         return GroupedData(jgd, self)
 
+    def unpivot(
+        self,
+        ids: Optional[Union["ColumnOrName", List["ColumnOrName"], Tuple["ColumnOrName", ...]]] = None,
+        values: Optional[Union["ColumnOrName", List["ColumnOrName"], Tuple["ColumnOrName", ...]]] = None,
+        variableColumnName: Optional[str] = None,
+        valueColumnName: Optional[str] = None,
+    ) -> "DataFrame":
+        """
+        Unpivot a DataFrame from wide format to long format, optionally leaving
+        identifier columns set. This is the reverse to `groupBy(...).pivot(...).agg(...)`,
+        except for the aggregation, which cannot be reversed.
+
+        This function is useful to massage a DataFrame into a format where some
+        columns are identifier columns ("ids"), while all other columns ("values")
+        are "unpivoted" to the rows, leaving just two non-id columns, named as given
+        by `variableColumnName` and `valueColumnName`.
+
+        When no "id" columns are given, the unpivoted DataFrame consists of only the
+        "variable" and "value" columns.
+
+        All "value" columns must share a least common data type. Unless they are the same data type,
+        all "value" columns are cast to the nearest common data type. For instance, types
+        `IntegerType` and `LongType` are cast to `LongType`, while `IntegerType` and `StringType`
+        do not have a common data type and `unpivot` fails.
+
+        :func:`groupby` is an alias for :func:`groupBy`.
+
+        .. versionadded:: 3.4.0
+
+        Parameters
+        ----------
+        ids : str, Column, tuple, list, optional
+            Column(s) to use as identifiers. Can be a single column or column name,
+            or a list for multiple columns.
+        values : str, Column, tuple, list, optional
+            Column(s) to unpivot. Can be a single column or column name, or a list
+            for multiple columns. If not specified or empty, uses all columns that
+            are not set as `ids`.
+        variableColumnName : scalar, default 'variable'
+            Name of the variable column. If None it uses 'variable'.
+        valueColumnName : scalar, default 'value'
+            Name of the value column. If None it uses 'value'.
+
+        Returns
+        -------
+        DataFrame
+            Unpivoted DataFrame.
+
+        Examples
+        --------
+        >>> df = spark.createDataFrame(
+        ...     [(1, 11, 1.1), (2, 12, 1.2)],
+        ...     ["id", "int", "double"],
+        ... )
+        >>> df.show()
+        +---+---+------+
+        | id|int|double|
+        +---+---+------+
+        |  1| 11|   1.1|
+        |  2| 12|   1.2|
+        +---+---+------+
+
+        >>> df.unpivot("id").show()
+        +---+--------+-----+
+        | id|variable|value|
+        +---+--------+-----+
+        |  1|     int| 11.0|
+        |  1|  double|  1.1|
+        |  2|     int| 12.0|
+        |  2|  double|  1.2|
+        +---+--------+-----+
+        """
+        def to_jcols(cols) -> JavaObject:
+            l = cols
+            if cols is None:
+                l = []
+            elif isinstance(cols, tuple):
+                l = list(cols)
+            elif not isinstance(cols, list):
+                l = [cols]
+            return self._jcols(*l)
+
+        if variableColumnName is None:
+            variableColumnName = "variable"

Review Comment:
   > Scala side doesn't have similar logic?
   
   Seems like Scala side doesn't have similar logic according to the [implementation](https://github.com/apache/spark/pull/36150/files) ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] EnricoMi commented on a diff in pull request #37304: [SPARK-39877][PySpark][WIP] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
EnricoMi commented on code in PR #37304:
URL: https://github.com/apache/spark/pull/37304#discussion_r930678850


##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -2127,6 +2127,15 @@ class Dataset[T] private[sql](
       valueColumnName: String): DataFrame =
     unpivot(ids, Array.empty, variableColumnName, valueColumnName)
 
+  /**
+   * Called from Python as Seq[Column] are easier to create via py4j than Array[Column].
+   */
+  private[sql] def _unpivot(ids: Seq[Column],

Review Comment:
   I'd prefer to get `PythonSQLUtils.toArray` return array of `Column`s but it returns array of `Object`. Then, this private method is not needed anymore. Looks like generic types do not work as expected.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on a diff in pull request #37304: [SPARK-39877][PySpark] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on code in PR #37304:
URL: https://github.com/apache/spark/pull/37304#discussion_r932840669


##########
python/pyspark/context.py:
##########
@@ -309,10 +309,7 @@ def _do_init(
         if sys.version_info[:2] < (3, 8):
             with warnings.catch_warnings():
                 warnings.simplefilter("once")
-                warnings.warn(
-                    "Python 3.7 support is deprecated in Spark 3.4.",
-                    FutureWarning
-                )
+                warnings.warn("Python 3.7 support is deprecated in Spark 3.4.", FutureWarning)

Review Comment:
   I encounter the same issue in other PR, maybe due to the `black` version. Let us revert this change



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zhengruifeng commented on a diff in pull request #37304: [SPARK-39877][PySpark] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on code in PR #37304:
URL: https://github.com/apache/spark/pull/37304#discussion_r932851912


##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -2127,6 +2127,15 @@ class Dataset[T] private[sql](
       valueColumnName: String): DataFrame =
     unpivot(ids, Array.empty, variableColumnName, valueColumnName)
 
+  /**
+   * Called from Python as Seq[Column] are easier to create via py4j than Array[Column].
+   */
+  private[sql] def _unpivot(ids: Seq[Column],

Review Comment:
   > And there are other private Python methods in Dataset (mapInPandas) and RelationalGroupedDataset (flatMapGroupsInPandas).
   
   they are also used in test



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] EnricoMi commented on a diff in pull request #37304: [SPARK-39877][PySpark] Add unpivot to PySpark DataFrame API

Posted by GitBox <gi...@apache.org>.
EnricoMi commented on code in PR #37304:
URL: https://github.com/apache/spark/pull/37304#discussion_r932977680


##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -2127,6 +2127,15 @@ class Dataset[T] private[sql](
       valueColumnName: String): DataFrame =
     unpivot(ids, Array.empty, variableColumnName, valueColumnName)
 
+  /**
+   * Called from Python as Seq[Column] are easier to create via py4j than Array[Column].
+   */
+  private[sql] def _unpivot(ids: Seq[Column],

Review Comment:
   Renamed `private[sql] def _unpivot` to `private[sql] def unpivot`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org