You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/05/25 05:56:02 UTC
[GitHub] [spark] zhengruifeng opened a new pull request, #36660: [SPARK-39284][PS] Implement Groupby.mad
zhengruifeng opened a new pull request, #36660:
URL: https://github.com/apache/spark/pull/36660
### What changes were proposed in this pull request?
Implement Groupby.mad
### Why are the changes needed?
to increase pandas api coverage
### Does this PR introduce _any_ user-facing change?
yes
```
In [6]: pdf = pd.DataFrame({"A": [1, 2, 2, 1, 1], "B": [3, 2, 3, 9, 0], "C": [3, 4, 13, -14, 9]})
In [7]: psdf = ps.from_pandas(pdf)
In [8]: pdf.groupby("A")[["B", "C"]].mad()
Out[8]:
B C
A
1 3.333333 8.888889
2 0.500000 4.500000
In [9]: psdf.groupby("A")[["B", "C"]].mad()
Out[9]:
B C
A
1 3.333333 8.888889
2 0.500000 4.500000
In [10]: pdf.B.groupby(pdf.A).mad()
Out[10]:
A
1 3.333333
2 0.500000
Name: B, dtype: float64
In [11]: psdf.B.groupby(psdf.A).mad()
Out[11]:
A
1 3.333333
2 0.500000
Name: B, dtype: float64
```
### How was this patch tested?
added ut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #36660: [SPARK-39284][PS] Implement Groupby.mad
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on code in PR #36660:
URL: https://github.com/apache/spark/pull/36660#discussion_r881246146
##########
python/pyspark/pandas/groupby.py:
##########
@@ -759,6 +759,99 @@ def skew(scol: Column) -> Column:
bool_to_numeric=True,
)
+ # TODO: 'axis', 'skipna', 'level' parameter should be implemented.
+ def mad(self) -> FrameLike:
+ """
+ Compute mean absolute deviation of groups, excluding missing values.
+
+ .. versionadded:: 3.4.0
+
+ Examples
+ --------
+ >>> df = ps.DataFrame({"A": [1, 2, 1, 1], "B": [True, False, False, True],
+ ... "C": [3, 4, 3, 4], "D": ["a", "b", "b", "a"]})
+
+ >>> df.groupby("A").mad()
+ B C
+ A
+ 1 0.444444 0.444444
+ 2 0.000000 0.000000
+
+ >>> df.B.groupby(df.A).mad()
+ A
+ 1 0.444444
+ 2 0.000000
+ Name: B, dtype: float64
+
+ See Also
+ --------
+ pyspark.pandas.Series.groupby
+ pyspark.pandas.DataFrame.groupby
+ """
+
+ groupkey_names = [SPARK_INDEX_NAME_FORMAT(i) for i in range(len(self._groupkeys))]
+ groupkey_scols = [s.alias(name) for s, name in zip(self._groupkeys_scols, groupkey_names)]
+
+ agg_columns = []
+ for psser in self._agg_columns:
+ if isinstance(psser.spark.data_type, BooleanType):
+ agg_columns.append(psser.astype(int))
+ elif isinstance(psser.spark.data_type, NumericType):
+ agg_columns.append(psser)
+
+ sdf = self._psdf._internal.spark_frame.select(
+ *groupkey_scols, *[psser.spark.column for psser in agg_columns]
+ )
+
+ internal = InternalFrame(
+ spark_frame=sdf,
+ index_spark_columns=[scol_for(sdf, col) for col in groupkey_names],
+ index_names=[psser._column_label for psser in self._groupkeys],
+ index_fields=[
+ psser._internal.data_fields[0].copy(name=name)
+ for psser, name in zip(self._groupkeys, groupkey_names)
+ ],
+ data_spark_columns=[
+ scol_for(sdf, psser._internal.data_spark_column_names[0]) for psser in agg_columns
+ ],
+ column_labels=[psser._column_label for psser in agg_columns],
+ data_fields=[psser._internal.data_fields[0] for psser in agg_columns],
+ column_label_names=self._psdf._internal.column_label_names,
+ )
+ psdf: DataFrame = DataFrame(internal)
+
+ if len(psdf._internal.column_labels) > 0:
Review Comment:
it is not able to directly use `self._reduce_for_stat_function` for `mad` due to `It is not allowed to use a window function inside an aggregate function`.
The implement here refer to `_reduce_for_stat_function`, but has some changes here (apply window function to compute the `mean` for each group)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a diff in pull request #36660: [SPARK-39284][PS] Implement Groupby.mad
Posted by GitBox <gi...@apache.org>.
itholic commented on code in PR #36660:
URL: https://github.com/apache/spark/pull/36660#discussion_r882216598
##########
python/pyspark/pandas/groupby.py:
##########
@@ -759,6 +759,99 @@ def skew(scol: Column) -> Column:
bool_to_numeric=True,
)
+ # TODO: 'axis', 'skipna', 'level' parameter should be implemented.
+ def mad(self) -> FrameLike:
+ """
+ Compute mean absolute deviation of groups, excluding missing values.
+
+ .. versionadded:: 3.4.0
+
+ Examples
+ --------
+ >>> df = ps.DataFrame({"A": [1, 2, 1, 1], "B": [True, False, False, True],
+ ... "C": [3, 4, 3, 4], "D": ["a", "b", "b", "a"]})
+
+ >>> df.groupby("A").mad()
+ B C
+ A
+ 1 0.444444 0.444444
+ 2 0.000000 0.000000
+
+ >>> df.B.groupby(df.A).mad()
+ A
+ 1 0.444444
+ 2 0.000000
+ Name: B, dtype: float64
+
+ See Also
+ --------
+ pyspark.pandas.Series.groupby
+ pyspark.pandas.DataFrame.groupby
+ """
+
+ groupkey_names = [SPARK_INDEX_NAME_FORMAT(i) for i in range(len(self._groupkeys))]
+ groupkey_scols = [s.alias(name) for s, name in zip(self._groupkeys_scols, groupkey_names)]
+
+ agg_columns = []
+ for psser in self._agg_columns:
+ if isinstance(psser.spark.data_type, BooleanType):
+ agg_columns.append(psser.astype(int))
+ elif isinstance(psser.spark.data_type, NumericType):
+ agg_columns.append(psser)
+
+ sdf = self._psdf._internal.spark_frame.select(
+ *groupkey_scols, *[psser.spark.column for psser in agg_columns]
+ )
+
+ internal = InternalFrame(
+ spark_frame=sdf,
+ index_spark_columns=[scol_for(sdf, col) for col in groupkey_names],
+ index_names=[psser._column_label for psser in self._groupkeys],
+ index_fields=[
+ psser._internal.data_fields[0].copy(name=name)
+ for psser, name in zip(self._groupkeys, groupkey_names)
+ ],
+ data_spark_columns=[
+ scol_for(sdf, psser._internal.data_spark_column_names[0]) for psser in agg_columns
+ ],
+ column_labels=[psser._column_label for psser in agg_columns],
+ data_fields=[psser._internal.data_fields[0] for psser in agg_columns],
+ column_label_names=self._psdf._internal.column_label_names,
+ )
+ psdf: DataFrame = DataFrame(internal)
+
+ if len(psdf._internal.column_labels) > 0:
Review Comment:
Yeah, it sounds reasonable to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #36660: [SPARK-39284][PS] Implement Groupby.mad
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on PR #36660:
URL: https://github.com/apache/spark/pull/36660#issuecomment-1138060258
https://github.com/apache/spark/pull/36547 provides two methods `_prepare_reduce` and `_prepare_return`, which can be used in this PR to simplify codes.
let us wait for https://github.com/apache/spark/pull/36547
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #36660: [SPARK-39284][PS] Implement Groupby.mad
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on PR #36660:
URL: https://github.com/apache/spark/pull/36660#issuecomment-1136882711
friendly ping @HyukjinKwon @itholic @xinrong-databricks @Yikun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #36660: [SPARK-39284][PS] Implement Groupby.mad
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on PR #36660:
URL: https://github.com/apache/spark/pull/36660#issuecomment-1146721683
merged to master, thanks all!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng closed pull request #36660: [SPARK-39284][PS] Implement Groupby.mad
Posted by GitBox <gi...@apache.org>.
zhengruifeng closed pull request #36660: [SPARK-39284][PS] Implement Groupby.mad
URL: https://github.com/apache/spark/pull/36660
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #36660: [SPARK-39284][PS] Implement Groupby.mad
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #36660:
URL: https://github.com/apache/spark/pull/36660#discussion_r888603854
##########
python/pyspark/pandas/groupby.py:
##########
@@ -759,6 +759,99 @@ def skew(scol: Column) -> Column:
bool_to_numeric=True,
)
+ # TODO: 'axis', 'skipna', 'level' parameter should be implemented.
+ def mad(self) -> FrameLike:
+ """
+ Compute mean absolute deviation of groups, excluding missing values.
+
+ .. versionadded:: 3.4.0
+
+ Examples
+ --------
+ >>> df = ps.DataFrame({"A": [1, 2, 1, 1], "B": [True, False, False, True],
+ ... "C": [3, 4, 3, 4], "D": ["a", "b", "b", "a"]})
+
+ >>> df.groupby("A").mad()
+ B C
+ A
+ 1 0.444444 0.444444
+ 2 0.000000 0.000000
+
+ >>> df.B.groupby(df.A).mad()
+ A
+ 1 0.444444
+ 2 0.000000
+ Name: B, dtype: float64
+
+ See Also
+ --------
+ pyspark.pandas.Series.groupby
+ pyspark.pandas.DataFrame.groupby
+ """
+
+ groupkey_names = [SPARK_INDEX_NAME_FORMAT(i) for i in range(len(self._groupkeys))]
+ groupkey_scols = [s.alias(name) for s, name in zip(self._groupkeys_scols, groupkey_names)]
+
+ agg_columns = []
+ for psser in self._agg_columns:
+ if isinstance(psser.spark.data_type, BooleanType):
+ agg_columns.append(psser.astype(int))
+ elif isinstance(psser.spark.data_type, NumericType):
+ agg_columns.append(psser)
+
+ sdf = self._psdf._internal.spark_frame.select(
+ *groupkey_scols, *[psser.spark.column for psser in agg_columns]
+ )
+
+ internal = InternalFrame(
+ spark_frame=sdf,
+ index_spark_columns=[scol_for(sdf, col) for col in groupkey_names],
+ index_names=[psser._column_label for psser in self._groupkeys],
+ index_fields=[
+ psser._internal.data_fields[0].copy(name=name)
+ for psser, name in zip(self._groupkeys, groupkey_names)
+ ],
+ data_spark_columns=[
+ scol_for(sdf, psser._internal.data_spark_column_names[0]) for psser in agg_columns
+ ],
+ column_labels=[psser._column_label for psser in agg_columns],
+ data_fields=[psser._internal.data_fields[0] for psser in agg_columns],
+ column_label_names=self._psdf._internal.column_label_names,
+ )
+ psdf: DataFrame = DataFrame(internal)
+
+ if len(psdf._internal.column_labels) > 0:
Review Comment:
Maybe lets add some comments here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #36660: [SPARK-39284][PS] Implement Groupby.mad
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on PR #36660:
URL: https://github.com/apache/spark/pull/36660#issuecomment-1141607627
cc @HyukjinKwon I think this PR is ready
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a diff in pull request #36660: [SPARK-39284][PS] Implement Groupby.mad
Posted by GitBox <gi...@apache.org>.
Yikun commented on code in PR #36660:
URL: https://github.com/apache/spark/pull/36660#discussion_r887650580
##########
python/pyspark/pandas/groupby.py:
##########
@@ -753,6 +753,75 @@ def skew(self) -> FrameLike:
bool_to_numeric=True,
)
+ # TODO: 'axis', 'skipna', 'level' parameter should be implemented.
+ def mad(self) -> FrameLike:
+ """
+ Compute mean absolute deviation of groups, excluding missing values.
+
+ .. versionadded:: 3.4.0
+
+ Examples
+ --------
+ >>> df = ps.DataFrame({"A": [1, 2, 1, 1], "B": [True, False, False, True],
+ ... "C": [3, 4, 3, 4], "D": ["a", "b", "b", "a"]})
+
+ >>> df.groupby("A").mad()
+ B C
+ A
+ 1 0.444444 0.444444
+ 2 0.000000 0.000000
+
+ >>> df.B.groupby(df.A).mad()
+ A
+ 1 0.444444
+ 2 0.000000
+ Name: B, dtype: float64
+
+ See Also
+ --------
+ pyspark.pandas.Series.groupby
+ pyspark.pandas.DataFrame.groupby
+ """
+ groupkey_names = [SPARK_INDEX_NAME_FORMAT(i) for i in range(len(self._groupkeys))]
+ internal, agg_columns, sdf = self._prepare_reduce(
+ groupkey_names=groupkey_names,
+ accepted_spark_types=(NumericType, BooleanType),
+ bool_to_numeric=False,
+ )
+ psdf: DataFrame = DataFrame(internal)
+
+ if len(psdf._internal.column_labels) > 0:
+ window = Window.partitionBy(groupkey_names).rowsBetween(
+ Window.unboundedPreceding, Window.unboundedFollowing
+ )
+ new_agg_scols = {}
+ new_stat_scols = []
+ for agg_column in agg_columns:
+ agg_column_name = agg_column._internal.data_spark_column_names[0]
+ new_agg_column_name = "__tmp_agg_col_{}__".format(agg_column_name)
Review Comment:
```suggestion
new_agg_column_name = verify_temp_column_name(
psdf._internal.spark_frame, "__tmp_agg_col_{}__".format(agg_column_name)
)
```
Maybe add a `verify_temp_column_name` check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #36660: [SPARK-39284][PS] Implement Groupby.mad
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #36660:
URL: https://github.com/apache/spark/pull/36660#discussion_r888603267
##########
python/pyspark/pandas/groupby.py:
##########
@@ -805,7 +874,7 @@ def all(self, skipna: bool = True) -> FrameLike:
5 False
"""
groupkey_names = [SPARK_INDEX_NAME_FORMAT(i) for i in range(len(self._groupkeys))]
- internal, sdf = self._prepare_reduce(groupkey_names)
+ internal, _, sdf = self._prepare_reduce(groupkey_names)
Review Comment:
Hm, I can't follow this change. `_prepare_reduce` seems returning internal frame and dataframe. what's the one in the middle?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org