You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/04/11 19:51:36 UTC
[GitHub] [spark] xinrong-databricks opened a new pull request, #36148: Implement `numeric_only` parameter of `GroupBy.max/min`
xinrong-databricks opened a new pull request, #36148:
URL: https://github.com/apache/spark/pull/36148
### What changes were proposed in this pull request?
Implement `numeric_only` parameter of `GroupBy.max/min`.
### Why are the changes needed?
To reach parity with pandas API.
### Does this PR introduce _any_ user-facing change?
Yes. `numeric_only` parameter is supported in `GroupBy.max/min`.
### How was this patch tested?
Unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a diff in pull request #36148: [SPARK-38880][PYTHON] Implement `numeric_only` parameter of `GroupBy.max/min`
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on code in PR #36148:
URL: https://github.com/apache/spark/pull/36148#discussion_r850625271
##########
python/pyspark/pandas/tests/test_groupby.py:
##########
@@ -1242,6 +1242,54 @@ def test_rank(self):
pdf.groupby([("x", "a"), ("x", "b")]).rank().sort_index(),
)
+ def test_min(self):
+ pdf = pd.DataFrame(
+ {
+ "A": [1, 2, 1, 2],
+ "B": [3, 4, 4, 3],
Review Comment:
Sounds good! Adjusted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #36148: [SPARK-38880][PYTHON] Implement `numeric_only` parameter of `GroupBy.max/min`
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #36148: [SPARK-38880][PYTHON] Implement `numeric_only` parameter of `GroupBy.max/min`
URL: https://github.com/apache/spark/pull/36148
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a diff in pull request #36148: [SPARK-38880][PYTHON] Implement `numeric_only` parameter of `GroupBy.max/min`
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on code in PR #36148:
URL: https://github.com/apache/spark/pull/36148#discussion_r850622966
##########
python/pyspark/pandas/groupby.py:
##########
@@ -426,16 +427,45 @@ def last(self) -> FrameLike:
lambda col: F.last(col, ignorenulls=True), only_numeric=False
)
- def max(self) -> FrameLike:
+ def max(self, numeric_only: Optional[bool] = False) -> FrameLike:
"""
Compute max of group values.
+ Parameters
+ ----------
+ numeric_only : bool, default False
+ Include only float, int, boolean columns. If None, will attempt to use
+ everything, then use only numeric data.
+
+ .. versionadded:: 3.4.0
+
See Also
--------
pyspark.pandas.Series.groupby
pyspark.pandas.DataFrame.groupby
+
+ Examples
+ --------
+ >>> df = ps.DataFrame({"A": [1, 2, 1, 2], "B": [True, False, False, True],
+ ... "C": [3, 4, 3, 4], "D": ["a", "b", "b", "a"]})
+
+ >>> df.groupby("A").max().sort_index()
+ B C D
+ A
+ 1 True 3 b
+ 2 True 4 b
Review Comment:
Thank you :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on pull request #36148: [SPARK-38880][PYTHON] Implement `numeric_only` parameter of `GroupBy.max/min`
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on PR #36148:
URL: https://github.com/apache/spark/pull/36148#issuecomment-1101570722
Thanks @itholic !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a diff in pull request #36148: [SPARK-38880][PYTHON] Implement `numeric_only` parameter of `GroupBy.max/min`
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on code in PR #36148:
URL: https://github.com/apache/spark/pull/36148#discussion_r850624426
##########
python/pyspark/pandas/groupby.py:
##########
@@ -2573,15 +2631,30 @@ def stat_function(col: Column) -> Column:
return self._reduce_for_stat_function(stat_function, only_numeric=numeric_only)
def _reduce_for_stat_function(
- self, sfun: Callable[[Column], Column], only_numeric: bool
+ self,
+ sfun: Callable[[Column], Column],
+ only_numeric: Optional[bool] = None,
+ bool_as_numeric: bool = False,
) -> FrameLike:
+ """Apply an aggregate function `sfun` per column and reduce to a FrameLike.
+
+ Parameters
+ ----------
+ sfun : The aggregate function to apply per column
+ only_numeric: If True, only numeric columns are involved
+ bool_as_numeric: If True, boolean columns are seen as numeric columns (following pandas)
+ """
groupkey_names = [SPARK_INDEX_NAME_FORMAT(i) for i in range(len(self._groupkeys))]
groupkey_scols = [s.alias(name) for s, name in zip(self._groupkeys_scols, groupkey_names)]
agg_columns = [
psser
for psser in self._agg_columns
- if isinstance(psser.spark.data_type, NumericType) or not only_numeric
+ if (
+ isinstance(psser.spark.data_type, NumericType)
+ or (isinstance(psser.spark.data_type, BooleanType) and bool_as_numeric)
Review Comment:
Actually, that's a pretty good idea. Adjusted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a diff in pull request #36148: [SPARK-38880][PYTHON] Implement `numeric_only` parameter of `GroupBy.max/min`
Posted by GitBox <gi...@apache.org>.
Yikun commented on code in PR #36148:
URL: https://github.com/apache/spark/pull/36148#discussion_r850412230
##########
python/pyspark/pandas/tests/test_groupby.py:
##########
@@ -1242,6 +1242,54 @@ def test_rank(self):
pdf.groupby([("x", "a"), ("x", "b")]).rank().sort_index(),
)
+ def test_min(self):
+ pdf = pd.DataFrame(
+ {
+ "A": [1, 2, 1, 2],
+ "B": [3, 4, 4, 3],
Review Comment:
```suggestion
"B": [3.1, 4.1, 4.1, 3.1],
```
nit: we might easy to cover `float, int, boolean columns` by change this
##########
python/pyspark/pandas/tests/test_groupby.py:
##########
@@ -1242,6 +1242,54 @@ def test_rank(self):
pdf.groupby([("x", "a"), ("x", "b")]).rank().sort_index(),
)
+ def test_min(self):
+ pdf = pd.DataFrame(
+ {
+ "A": [1, 2, 1, 2],
+ "B": [3, 4, 4, 3],
+ "C": ["a", "b", "b", "a"],
+ "D": [True, False, False, True],
+ }
+ )
+ psdf = ps.from_pandas(pdf)
+ for p_groupby_obj, ps_groupby_obj in [
+ (pdf.groupby("A"), psdf.groupby("A")),
+ (pdf.groupby("A")[["C"]], psdf.groupby("A")[["C"]]),
+ ]:
+ self.assert_eq(p_groupby_obj.min().sort_index(), ps_groupby_obj.min().sort_index())
+ self.assert_eq(
+ p_groupby_obj.min(numeric_only=None).sort_index(),
+ ps_groupby_obj.min(numeric_only=None).sort_index(),
+ )
+ self.assert_eq(
+ p_groupby_obj.min(numeric_only=True).sort_index(),
+ ps_groupby_obj.min(numeric_only=True).sort_index(),
+ )
+
+ def test_max(self):
+ pdf = pd.DataFrame(
+ {
+ "A": [1, 2, 1, 2],
+ "B": [3, 4, 4, 3],
Review Comment:
```suggestion
"B": [3.1, 4.1, 4.1, 3.1],
```
same
##########
python/pyspark/pandas/groupby.py:
##########
@@ -2573,15 +2631,30 @@ def stat_function(col: Column) -> Column:
return self._reduce_for_stat_function(stat_function, only_numeric=numeric_only)
def _reduce_for_stat_function(
- self, sfun: Callable[[Column], Column], only_numeric: bool
+ self,
+ sfun: Callable[[Column], Column],
+ only_numeric: Optional[bool] = None,
+ bool_as_numeric: bool = False,
) -> FrameLike:
+ """Apply an aggregate function `sfun` per column and reduce to a FrameLike.
+
+ Parameters
+ ----------
+ sfun : The aggregate function to apply per column
+ only_numeric: If True, only numeric columns are involved
+ bool_as_numeric: If True, boolean columns are seen as numeric columns (following pandas)
+ """
groupkey_names = [SPARK_INDEX_NAME_FORMAT(i) for i in range(len(self._groupkeys))]
groupkey_scols = [s.alias(name) for s, name in zip(self._groupkeys_scols, groupkey_names)]
agg_columns = [
psser
for psser in self._agg_columns
- if isinstance(psser.spark.data_type, NumericType) or not only_numeric
+ if (
+ isinstance(psser.spark.data_type, NumericType)
+ or (isinstance(psser.spark.data_type, BooleanType) and bool_as_numeric)
Review Comment:
```suggestion
or (bool_as_numeric and isinstance(psser.spark.data_type, BooleanType))
```
super nits: this might reduce isinstance cost by short-circuiting, fine for me if you think it's no need to change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a diff in pull request #36148: [SPARK-38880][PYTHON] Implement `numeric_only` parameter of `GroupBy.max/min`
Posted by GitBox <gi...@apache.org>.
itholic commented on code in PR #36148:
URL: https://github.com/apache/spark/pull/36148#discussion_r850082074
##########
python/pyspark/pandas/groupby.py:
##########
@@ -426,16 +427,45 @@ def last(self) -> FrameLike:
lambda col: F.last(col, ignorenulls=True), only_numeric=False
)
- def max(self) -> FrameLike:
+ def max(self, numeric_only: Optional[bool] = False) -> FrameLike:
"""
Compute max of group values.
+ Parameters
+ ----------
+ numeric_only : bool, default False
+ Include only float, int, boolean columns. If None, will attempt to use
+ everything, then use only numeric data.
+
+ .. versionadded:: 3.4.0
+
See Also
--------
pyspark.pandas.Series.groupby
pyspark.pandas.DataFrame.groupby
+
+ Examples
+ --------
+ >>> df = ps.DataFrame({"A": [1, 2, 1, 2], "B": [True, False, False, True],
+ ... "C": [3, 4, 3, 4], "D": ["a", "b", "b", "a"]})
+
+ >>> df.groupby("A").max().sort_index()
+ B C D
+ A
+ 1 True 3 b
+ 2 True 4 b
Review Comment:
Thanks for also adding the normal case testing!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a diff in pull request #36148: [SPARK-38880][PYTHON] Implement `numeric_only` parameter of `GroupBy.max/min`
Posted by GitBox <gi...@apache.org>.
ueshin commented on code in PR #36148:
URL: https://github.com/apache/spark/pull/36148#discussion_r852312830
##########
python/pyspark/pandas/groupby.py:
##########
@@ -2573,15 +2631,30 @@ def stat_function(col: Column) -> Column:
return self._reduce_for_stat_function(stat_function, only_numeric=numeric_only)
def _reduce_for_stat_function(
- self, sfun: Callable[[Column], Column], only_numeric: bool
+ self,
+ sfun: Callable[[Column], Column],
+ only_numeric: Optional[bool] = None,
+ bool_as_numeric: bool = False,
) -> FrameLike:
+ """Apply an aggregate function `sfun` per column and reduce to a FrameLike.
+
+ Parameters
+ ----------
+ sfun : The aggregate function to apply per column
+ only_numeric: If True, only numeric columns are involved
+ bool_as_numeric: If True, boolean columns are seen as numeric columns (following pandas)
+ """
groupkey_names = [SPARK_INDEX_NAME_FORMAT(i) for i in range(len(self._groupkeys))]
groupkey_scols = [s.alias(name) for s, name in zip(self._groupkeys_scols, groupkey_names)]
agg_columns = [
psser
for psser in self._agg_columns
- if isinstance(psser.spark.data_type, NumericType) or not only_numeric
+ if (
+ isinstance(psser.spark.data_type, NumericType)
+ or (isinstance(psser.spark.data_type, BooleanType) and bool_as_numeric)
Review Comment:
nit: In that case, we might also want to move `not only_numeric` to the first place?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on pull request #36148: [SPARK-38880][PYTHON] Implement `numeric_only` parameter of `GroupBy.max/min`
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on PR #36148:
URL: https://github.com/apache/spark/pull/36148#issuecomment-1098427518
@ueshin @HyukjinKwon @itholic Thanks!
Please refer to https://github.com/xinrong-databricks/spark/actions/workflows/build_and_test.yml for test results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #36148: [SPARK-38880][PYTHON] Implement `numeric_only` parameter of `GroupBy.max/min`
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #36148:
URL: https://github.com/apache/spark/pull/36148#issuecomment-1101919932
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on pull request #36148: [SPARK-38880][PYTHON] Implement `numeric_only` parameter of `GroupBy.max/min`
Posted by GitBox <gi...@apache.org>.
itholic commented on PR #36148:
URL: https://github.com/apache/spark/pull/36148#issuecomment-1100953205
https://github.com/xinrong-databricks/spark/actions/runs/2168476739 test passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org