You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/09/18 19:53:15 UTC
[GitHub] [spark] ayudovin opened a new pull request, #37923: [SPARK-40334] - Implement `GroupBy.prod`
ayudovin opened a new pull request, #37923:
URL: https://github.com/apache/spark/pull/37923
### What changes were proposed in this pull request?
Implement `GroupBy.prod`
### Why are the changes needed?
for API coverage
### Does this PR introduce _any_ user-facing change?
yes, the new API
```
df = ps.DataFrame({'A': [1, 1, 2, 1, 2],
'B': [np.nan, 2, 3, 4, 5],
'C': [1, 2, 1, 1, 2],
'D': [True, False, True, False, True]})
Groupby one column and return the prod of the remaining columns in
each group.
df.groupby('A').prod()
B C D
A
1 8.0 2 0
2 15.0 2 11
df.groupby('A').prod(min_count=3)
B C D
A
1 NaN 2 0
2 NaN NaN NaN
```
### How was this patch tested?
added UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #37923: [SPARK-40334][PS] Implement `GroupBy.prod`
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on code in PR #37923:
URL: https://github.com/apache/spark/pull/37923#discussion_r974780246
##########
python/pyspark/pandas/groupby.py:
##########
@@ -993,6 +994,101 @@ def nth(self, n: int) -> FrameLike:
return self._prepare_return(DataFrame(internal))
+ def prod(self, numeric_only: Optional[bool] = True, min_count: int = 0):
+ """
+ Compute prod of groups.
+
+ .. versionadded:: 3.4.0
+
+ Parameters
+ ----------
+ numeric_only : bool, default False
+ Include only float, int, boolean columns. If None, will attempt to use
+ everything, then use only numeric data.
+
+ min_count: int, default 0
+ The required number of valid values to perform the operation.
+ If fewer than min_count non-NA values are present the result will be NA.
+
+ Returns
+ -------
+ pyspark.pandas.Series or pyspark.pandas.DataFrame
Review Comment:
```suggestion
Series or DataFrame
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #37923: [SPARK-40334][PS] Implement `GroupBy.prod`
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on code in PR #37923:
URL: https://github.com/apache/spark/pull/37923#discussion_r977124585
##########
python/pyspark/pandas/groupby.py:
##########
@@ -993,6 +993,115 @@ def nth(self, n: int) -> FrameLike:
return self._prepare_return(DataFrame(internal))
+ def prod(self, numeric_only: Optional[bool] = True, min_count: int = 0) -> FrameLike:
+ """
+ Compute prod of groups.
+
+ .. versionadded:: 3.4.0
+
+ Parameters
+ ----------
+ numeric_only : bool, default False
+ Include only float, int, boolean columns. If None, will attempt to use
+ everything, then use only numeric data.
+
+ min_count: int, default 0
+ The required number of valid values to perform the operation.
+ If fewer than min_count non-NA values are present the result will be NA.
+
+ Returns
+ -------
+ Series or DataFrame
+ Computed prod of values within each group.
+
+ See Also
+ --------
+ pyspark.pandas.Series.groupby
+ pyspark.pandas.DataFrame.groupby
+
+ Examples
+ --------
+ >>> df = ps.DataFrame(
+ ... {
+ ... "A": [1, 1, 2, 1, 2],
+ ... "B": [np.nan, 2, 3, 4, 5],
+ ... "C": [1, 2, 1, 1, 2],
+ ... "D": [True, False, True, False, True],
+ ... }
+ ... )
+
+ Groupby one column and return the prod of the remaining columns in
+ each group.
+
+ >>> df.groupby('A').prod().sort_index()
+ B C D
+ A
+ 1 8.0 2 0
+ 2 15.0 2 1
+
+ >>> df.groupby('A').prod(min_count=3).sort_index()
+ B C D
+ A
+ 1 NaN 2.0 0.0
+ 2 NaN NaN NaN
+ """
+
+ self._validate_agg_columns(numeric_only=numeric_only, function_name="prod")
+
+ groupkey_names = [SPARK_INDEX_NAME_FORMAT(i) for i in range(len(self._groupkeys))]
+ internal, agg_columns, sdf = self._prepare_reduce(
+ groupkey_names=groupkey_names,
+ accepted_spark_types=(NumericType, BooleanType),
+ bool_to_numeric=True,
+ )
+
+ psdf: DataFrame = DataFrame(internal)
+ if len(psdf._internal.column_labels) > 0:
+
+ stat_exprs = []
+ for label in psdf._internal.column_labels:
+ label_name = label[0]
+ tmp_count_column_name = verify_temp_column_name(
+ sdf, "__tmp_%s_count_col__" % label_name
+ )
+ psser = psdf._psser_for(label)
+ column = psser._dtype_op.nan_to_null(psser).spark.column
+ data_type = psser.spark.data_type
+
+ if isinstance(data_type, IntegralType):
+ stat_exprs.append(F.product(column).cast(data_type).alias(label_name))
+ else:
+ stat_exprs.append(F.product(column).alias(label_name))
+
+ if min_count > 0:
+ stat_exprs.append(F.count(column).alias(tmp_count_column_name))
+
+ sdf = sdf.groupby(*groupkey_names).agg(*stat_exprs)
Review Comment:
Can we simplify the logic like this?
```suggestion
for label in psdf._internal.column_labels:
psser = psdf._psser_for(label)
scol = psser._dtype_op.nan_to_null(psser).spark.column
prod_scol = F.product(scol)
if min_count > 0:
prod_scol = F.when(F.count(scol) < min_count, F.lit(None)).otherwise(F.product(scol))
else:
prod_scol = F.product(scol)
stat_exprs.append(
prod_scol.alias(psser._internal.data_spark_column_names[0])
)
sdf = sdf.groupby(*groupkey_names).agg(*stat_exprs)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a diff in pull request #37923: [SPARK-40334][PS] Implement `GroupBy.prod`
Posted by GitBox <gi...@apache.org>.
Yikun commented on code in PR #37923:
URL: https://github.com/apache/spark/pull/37923#discussion_r973941663
##########
python/pyspark/pandas/groupby.py:
##########
@@ -993,6 +993,98 @@ def nth(self, n: int) -> FrameLike:
return self._prepare_return(DataFrame(internal))
+ def prod(self, numeric_only: Optional[bool] = True, min_count: int = 0):
+ """
+ Compute prod of groups.
+
+ Parameters
+ ----------
+ numeric_only : bool, default False
+ Include only float, int, boolean columns. If None, will attempt to use
+ everything, then use only numeric data.
+
+ min_count: int, default 0
+ The required number of valid values to perform the operation.
+ If fewer than min_count non-NA values are present the result will be NA.
+
+ .. versionadded:: 3.4.0
Review Comment:
You could also generate and see final doc in local:
https://spark.apache.org/docs/latest/api/python/development/contributing.html#contributing-documentation-changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a diff in pull request #37923: [SPARK-40334][PS] Implement `GroupBy.prod`
Posted by GitBox <gi...@apache.org>.
Yikun commented on code in PR #37923:
URL: https://github.com/apache/spark/pull/37923#discussion_r977130360
##########
python/pyspark/pandas/groupby.py:
##########
@@ -993,6 +993,115 @@ def nth(self, n: int) -> FrameLike:
return self._prepare_return(DataFrame(internal))
+ def prod(self, numeric_only: Optional[bool] = True, min_count: int = 0) -> FrameLike:
+ """
+ Compute prod of groups.
+
+ .. versionadded:: 3.4.0
+
+ Parameters
+ ----------
+ numeric_only : bool, default False
+ Include only float, int, boolean columns. If None, will attempt to use
+ everything, then use only numeric data.
+
+ min_count: int, default 0
+ The required number of valid values to perform the operation.
+ If fewer than min_count non-NA values are present the result will be NA.
+
+ Returns
+ -------
+ Series or DataFrame
+ Computed prod of values within each group.
+
+ See Also
+ --------
+ pyspark.pandas.Series.groupby
+ pyspark.pandas.DataFrame.groupby
+
+ Examples
+ --------
+ >>> df = ps.DataFrame(
Review Comment:
```suggestion
>>> import numpy as np
>>> df = ps.DataFrame(
```
Let's make it as self-contained examples, it means users can copy/paste to python shell directly, see also: SPARK-40005
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin commented on a diff in pull request #37923: [SPARK-40334][PS] Implement `GroupBy.prod`
Posted by GitBox <gi...@apache.org>.
ayudovin commented on code in PR #37923:
URL: https://github.com/apache/spark/pull/37923#discussion_r977574999
##########
python/pyspark/pandas/groupby.py:
##########
@@ -993,6 +993,115 @@ def nth(self, n: int) -> FrameLike:
return self._prepare_return(DataFrame(internal))
+ def prod(self, numeric_only: Optional[bool] = True, min_count: int = 0) -> FrameLike:
+ """
+ Compute prod of groups.
+
+ .. versionadded:: 3.4.0
+
+ Parameters
+ ----------
+ numeric_only : bool, default False
+ Include only float, int, boolean columns. If None, will attempt to use
+ everything, then use only numeric data.
+
+ min_count: int, default 0
+ The required number of valid values to perform the operation.
+ If fewer than min_count non-NA values are present the result will be NA.
+
+ Returns
+ -------
+ Series or DataFrame
+ Computed prod of values within each group.
+
+ See Also
+ --------
+ pyspark.pandas.Series.groupby
+ pyspark.pandas.DataFrame.groupby
+
+ Examples
+ --------
+ >>> df = ps.DataFrame(
+ ... {
+ ... "A": [1, 1, 2, 1, 2],
+ ... "B": [np.nan, 2, 3, 4, 5],
+ ... "C": [1, 2, 1, 1, 2],
+ ... "D": [True, False, True, False, True],
+ ... }
+ ... )
+
+ Groupby one column and return the prod of the remaining columns in
+ each group.
+
+ >>> df.groupby('A').prod().sort_index()
+ B C D
+ A
+ 1 8.0 2 0
+ 2 15.0 2 1
+
+ >>> df.groupby('A').prod(min_count=3).sort_index()
+ B C D
+ A
+ 1 NaN 2.0 0.0
+ 2 NaN NaN NaN
+ """
+
+ self._validate_agg_columns(numeric_only=numeric_only, function_name="prod")
+
+ groupkey_names = [SPARK_INDEX_NAME_FORMAT(i) for i in range(len(self._groupkeys))]
+ internal, agg_columns, sdf = self._prepare_reduce(
+ groupkey_names=groupkey_names,
+ accepted_spark_types=(NumericType, BooleanType),
+ bool_to_numeric=True,
+ )
+
+ psdf: DataFrame = DataFrame(internal)
+ if len(psdf._internal.column_labels) > 0:
+
+ stat_exprs = []
+ for label in psdf._internal.column_labels:
+ label_name = label[0]
+ tmp_count_column_name = verify_temp_column_name(
+ sdf, "__tmp_%s_count_col__" % label_name
+ )
+ psser = psdf._psser_for(label)
+ column = psser._dtype_op.nan_to_null(psser).spark.column
+ data_type = psser.spark.data_type
+
+ if isinstance(data_type, IntegralType):
Review Comment:
What do you mean by `always returns LongType`?
Always cast the result of F.product() to LongType?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin commented on a diff in pull request #37923: [SPARK-40334][PS] Implement `GroupBy.prod`
Posted by GitBox <gi...@apache.org>.
ayudovin commented on code in PR #37923:
URL: https://github.com/apache/spark/pull/37923#discussion_r978737112
##########
python/pyspark/pandas/groupby.py:
##########
@@ -993,6 +993,106 @@ def nth(self, n: int) -> FrameLike:
return self._prepare_return(DataFrame(internal))
+ def prod(self, numeric_only: Optional[bool] = True, min_count: int = 0) -> FrameLike:
+ """
+ Compute prod of groups.
+
+ .. versionadded:: 3.4.0
+
+ Parameters
+ ----------
+ numeric_only : bool, default False
+ Include only float, int, boolean columns. If None, will attempt to use
+ everything, then use only numeric data.
+
+ min_count: int, default 0
+ The required number of valid values to perform the operation.
+ If fewer than min_count non-NA values are present the result will be NA.
+
+ Returns
+ -------
+ Series or DataFrame
+ Computed prod of values within each group.
+
+ See Also
+ --------
+ pyspark.pandas.Series.groupby
+ pyspark.pandas.DataFrame.groupby
+
+ Examples
+ --------
+ >>> import numpy as np
+ >>> df = ps.DataFrame(
+ ... {
+ ... "A": [1, 1, 2, 1, 2],
+ ... "B": [np.nan, 2, 3, 4, 5],
+ ... "C": [1, 2, 1, 1, 2],
+ ... "D": [True, False, True, False, True],
+ ... }
+ ... )
+
+ Groupby one column and return the prod of the remaining columns in
+ each group.
+
+ >>> df.groupby('A').prod().sort_index()
+ B C D
+ A
+ 1 8.0 2 0
+ 2 15.0 2 1
+
+ >>> df.groupby('A').prod(min_count=3).sort_index()
+ B C D
+ A
+ 1 NaN 2.0 0.0
+ 2 NaN NaN NaN
+ """
+
+ self._validate_agg_columns(numeric_only=numeric_only, function_name="prod")
+
+ groupkey_names = [SPARK_INDEX_NAME_FORMAT(i) for i in range(len(self._groupkeys))]
+ internal, agg_columns, sdf = self._prepare_reduce(
+ groupkey_names=groupkey_names,
+ accepted_spark_types=(NumericType, BooleanType),
+ bool_to_numeric=True,
+ )
+
+ psdf: DataFrame = DataFrame(internal)
+ if len(psdf._internal.column_labels) > 0:
+
+ stat_exprs = []
+ for label in psdf._internal.column_labels:
+ psser = psdf._psser_for(label)
+ column = psser._dtype_op.nan_to_null(psser).spark.column
+ data_type = psser.spark.data_type
+ aggregating = (
+ F.product(column).cast(data_type)
Review Comment:
Yep, that makes sense, I added it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a diff in pull request #37923: [SPARK-40334][PS] Implement `GroupBy.prod`
Posted by GitBox <gi...@apache.org>.
Yikun commented on code in PR #37923:
URL: https://github.com/apache/spark/pull/37923#discussion_r974808692
##########
python/pyspark/pandas/groupby.py:
##########
@@ -993,6 +994,101 @@ def nth(self, n: int) -> FrameLike:
return self._prepare_return(DataFrame(internal))
+ def prod(self, numeric_only: Optional[bool] = True, min_count: int = 0):
Review Comment:
```suggestion
def prod(self, numeric_only: Optional[bool] = True, min_count: int = 0) -> FrameLike:
```
##########
python/pyspark/pandas/groupby.py:
##########
@@ -18,7 +18,6 @@
"""
A wrapper for GroupedData to behave similar to pandas GroupBy.
"""
-
Review Comment:
unrelated change
##########
python/pyspark/pandas/groupby.py:
##########
@@ -993,6 +994,101 @@ def nth(self, n: int) -> FrameLike:
return self._prepare_return(DataFrame(internal))
+ def prod(self, numeric_only: Optional[bool] = True, min_count: int = 0):
+ """
+ Compute prod of groups.
+
+ .. versionadded:: 3.4.0
+
+ Parameters
+ ----------
+ numeric_only : bool, default False
+ Include only float, int, boolean columns. If None, will attempt to use
+ everything, then use only numeric data.
+
+ min_count: int, default 0
+ The required number of valid values to perform the operation.
+ If fewer than min_count non-NA values are present the result will be NA.
+
+ Returns
+ -------
+ pyspark.pandas.Series or pyspark.pandas.DataFrame
+
+ See Also
+ --------
+ pyspark.pandas.Series.groupby
+ pyspark.pandas.DataFrame.groupby
+
+ Examples
+ --------
+ >>> df = ps.DataFrame({'A': [1, 1, 2, 1, 2],
+ ... 'B': [np.nan, 2, 3, 4, 5],
+ ... 'C': [1, 2, 1, 1, 2],
+ ... 'D': [True, False, True, False, True]})
+
+ Groupby one column and return the prod of the remaining columns in
+ each group.
+
+ >>> df.groupby('A').prod().sort_index()
+ B C D
+ A
+ 1 8.0 2 0
+ 2 15.0 2 1
+
+ >>> df.groupby('A').prod(min_count=3).sort_index()
+ B C D
+ A
+ 1 NaN 2.0 0.0
+ 2 NaN NaN NaN
+ """
+
+ self._validate_agg_columns(numeric_only=numeric_only, function_name="prod")
+
+ groupkey_names = [SPARK_INDEX_NAME_FORMAT(i) for i in range(len(self._groupkeys))]
+ internal, agg_columns, sdf = self._prepare_reduce(
+ groupkey_names=groupkey_names,
+ accepted_spark_types=(NumericType, BooleanType),
+ bool_to_numeric=True,
+ )
+
+ psdf: DataFrame = DataFrame(internal)
+ if len(psdf._internal.column_labels) > 0:
+ tmp_count_column = "__tmp_%s_count_col__"
Review Comment:
```suggestion
tmp_count_column = verify_temp_column_name(psdf, "__tmp_%s_count_col__")
```
You might want to verify column to aovid pontential column name conflict.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng closed pull request #37923: [SPARK-40334][PS] Implement `GroupBy.prod`
Posted by GitBox <gi...@apache.org>.
zhengruifeng closed pull request #37923: [SPARK-40334][PS] Implement `GroupBy.prod`
URL: https://github.com/apache/spark/pull/37923
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a diff in pull request #37923: [SPARK-40334][PS] Implement `GroupBy.prod`
Posted by GitBox <gi...@apache.org>.
itholic commented on code in PR #37923:
URL: https://github.com/apache/spark/pull/37923#discussion_r975910504
##########
python/pyspark/pandas/groupby.py:
##########
@@ -993,6 +993,105 @@ def nth(self, n: int) -> FrameLike:
return self._prepare_return(DataFrame(internal))
+ def prod(self, numeric_only: Optional[bool] = True, min_count: int = 0) -> FrameLike:
+ """
+ Compute prod of groups.
+
+ .. versionadded:: 3.4.0
+
+ Parameters
+ ----------
+ numeric_only : bool, default False
+ Include only float, int, boolean columns. If None, will attempt to use
+ everything, then use only numeric data.
+
+ min_count: int, default 0
+ The required number of valid values to perform the operation.
+ If fewer than min_count non-NA values are present the result will be NA.
+
+ Returns
+ -------
+ Series or DataFrame
+
+ See Also
+ --------
+ pyspark.pandas.Series.groupby
+ pyspark.pandas.DataFrame.groupby
+
+ Examples
+ --------
+ >>> df = ps.DataFrame({'A': [1, 1, 2, 1, 2],
+ ... 'B': [np.nan, 2, 3, 4, 5],
+ ... 'C': [1, 2, 1, 1, 2],
+ ... 'D': [True, False, True, False, True]})
Review Comment:
nit: formatting
```suggestion
>>> df = ps.DataFrame(
... {
... "A": [1, 1, 2, 1, 2],
... "B": [np.nan, 2, 3, 4, 5],
... "C": [1, 2, 1, 1, 2],
... "D": [True, False, True, False, True],
... }
... )
```
##########
python/pyspark/pandas/groupby.py:
##########
@@ -993,6 +993,105 @@ def nth(self, n: int) -> FrameLike:
return self._prepare_return(DataFrame(internal))
+ def prod(self, numeric_only: Optional[bool] = True, min_count: int = 0) -> FrameLike:
+ """
+ Compute prod of groups.
+
+ .. versionadded:: 3.4.0
+
+ Parameters
+ ----------
+ numeric_only : bool, default False
+ Include only float, int, boolean columns. If None, will attempt to use
+ everything, then use only numeric data.
+
+ min_count: int, default 0
+ The required number of valid values to perform the operation.
+ If fewer than min_count non-NA values are present the result will be NA.
+
+ Returns
+ -------
+ Series or DataFrame
+
+ See Also
+ --------
+ pyspark.pandas.Series.groupby
+ pyspark.pandas.DataFrame.groupby
+
+ Examples
+ --------
+ >>> df = ps.DataFrame({'A': [1, 1, 2, 1, 2],
+ ... 'B': [np.nan, 2, 3, 4, 5],
+ ... 'C': [1, 2, 1, 1, 2],
+ ... 'D': [True, False, True, False, True]})
+
+ Groupby one column and return the prod of the remaining columns in
+ each group.
+
+ >>> df.groupby('A').prod().sort_index()
+ B C D
+ A
+ 1 8.0 2 0
+ 2 15.0 2 1
+
+ >>> df.groupby('A').prod(min_count=3).sort_index()
+ B C D
+ A
+ 1 NaN 2.0 0.0
+ 2 NaN NaN NaN
+ """
+
+ self._validate_agg_columns(numeric_only=numeric_only, function_name="prod")
+
+ groupkey_names = [SPARK_INDEX_NAME_FORMAT(i) for i in range(len(self._groupkeys))]
+ internal, agg_columns, sdf = self._prepare_reduce(
+ groupkey_names=groupkey_names,
+ accepted_spark_types=(NumericType, BooleanType),
+ bool_to_numeric=True,
+ )
+
+ psdf: DataFrame = DataFrame(internal)
+ if len(psdf._internal.column_labels) > 0:
+
+ stat_exprs = []
+ for label in psdf._internal.column_labels:
+ tmp_count_column = verify_temp_column_name(sdf, "__tmp_%s_count_col__" % label[0])
+ psser = psdf._psser_for(label)
+ column = psser._dtype_op.nan_to_null(psser).spark.column
+ data_type = psser.spark.data_type
+
+ if isinstance(data_type, IntegralType):
+ stat_exprs.append(F.product(column).cast(data_type).alias(label[0]))
+ else:
+ stat_exprs.append(F.product(column).alias(label[0]))
Review Comment:
What about defining a `label[0]` as a variable since it's used in multiple places ?
##########
python/pyspark/pandas/groupby.py:
##########
@@ -993,6 +993,105 @@ def nth(self, n: int) -> FrameLike:
return self._prepare_return(DataFrame(internal))
+ def prod(self, numeric_only: Optional[bool] = True, min_count: int = 0) -> FrameLike:
+ """
+ Compute prod of groups.
+
+ .. versionadded:: 3.4.0
+
+ Parameters
+ ----------
+ numeric_only : bool, default False
+ Include only float, int, boolean columns. If None, will attempt to use
+ everything, then use only numeric data.
+
+ min_count: int, default 0
+ The required number of valid values to perform the operation.
+ If fewer than min_count non-NA values are present the result will be NA.
+
+ Returns
+ -------
+ Series or DataFrame
+
+ See Also
+ --------
+ pyspark.pandas.Series.groupby
+ pyspark.pandas.DataFrame.groupby
+
+ Examples
+ --------
+ >>> df = ps.DataFrame({'A': [1, 1, 2, 1, 2],
+ ... 'B': [np.nan, 2, 3, 4, 5],
+ ... 'C': [1, 2, 1, 1, 2],
+ ... 'D': [True, False, True, False, True]})
+
+ Groupby one column and return the prod of the remaining columns in
+ each group.
+
+ >>> df.groupby('A').prod().sort_index()
+ B C D
+ A
+ 1 8.0 2 0
+ 2 15.0 2 1
+
+ >>> df.groupby('A').prod(min_count=3).sort_index()
+ B C D
+ A
+ 1 NaN 2.0 0.0
+ 2 NaN NaN NaN
+ """
+
+ self._validate_agg_columns(numeric_only=numeric_only, function_name="prod")
+
+ groupkey_names = [SPARK_INDEX_NAME_FORMAT(i) for i in range(len(self._groupkeys))]
+ internal, agg_columns, sdf = self._prepare_reduce(
+ groupkey_names=groupkey_names,
+ accepted_spark_types=(NumericType, BooleanType),
+ bool_to_numeric=True,
+ )
+
+ psdf: DataFrame = DataFrame(internal)
+ if len(psdf._internal.column_labels) > 0:
+
+ stat_exprs = []
+ for label in psdf._internal.column_labels:
+ tmp_count_column = verify_temp_column_name(sdf, "__tmp_%s_count_col__" % label[0])
+ psser = psdf._psser_for(label)
+ column = psser._dtype_op.nan_to_null(psser).spark.column
+ data_type = psser.spark.data_type
+
+ if isinstance(data_type, IntegralType):
+ stat_exprs.append(F.product(column).cast(data_type).alias(label[0]))
+ else:
+ stat_exprs.append(F.product(column).alias(label[0]))
+
+ if min_count > 0:
+ stat_exprs.append(F.count(column).alias(tmp_count_column))
+
+ sdf = sdf.groupby(*groupkey_names).agg(*stat_exprs)
+
+ if min_count > 0:
+ for label in psdf._internal.column_labels:
+ tmp_count_column = "__tmp_%s_count_col__"
+ sdf = sdf.withColumn(
+ label[0],
+ F.when(
+ F.col(tmp_count_column % label[0]).__ge__(min_count), F.col(label[0])
Review Comment:
ditto ?
##########
python/pyspark/pandas/groupby.py:
##########
@@ -993,6 +993,105 @@ def nth(self, n: int) -> FrameLike:
return self._prepare_return(DataFrame(internal))
+ def prod(self, numeric_only: Optional[bool] = True, min_count: int = 0) -> FrameLike:
+ """
+ Compute prod of groups.
+
+ .. versionadded:: 3.4.0
+
+ Parameters
+ ----------
+ numeric_only : bool, default False
+ Include only float, int, boolean columns. If None, will attempt to use
+ everything, then use only numeric data.
+
+ min_count: int, default 0
+ The required number of valid values to perform the operation.
+ If fewer than min_count non-NA values are present the result will be NA.
+
+ Returns
+ -------
+ Series or DataFrame
Review Comment:
Let's add a sub-description as pandas does.
```suggestion
Series or DataFrame
Computed prod of values within each group.
```
##########
python/pyspark/pandas/groupby.py:
##########
@@ -1052,10 +1052,10 @@ def prod(self, numeric_only: Optional[bool] = True, min_count: int = 0) -> Frame
psdf: DataFrame = DataFrame(internal)
if len(psdf._internal.column_labels) > 0:
- tmp_count_column = verify_temp_column_name(psdf, "__tmp_%s_count_col__")
stat_exprs = []
for label in psdf._internal.column_labels:
+ tmp_count_column = verify_temp_column_name(sdf, "__tmp_%s_count_col__" % label[0])
Review Comment:
nit: use `tmp_count_column_name` instead to make it more explicit ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin commented on a diff in pull request #37923: [SPARK-40334][PS] Implement `GroupBy.prod`
Posted by GitBox <gi...@apache.org>.
ayudovin commented on code in PR #37923:
URL: https://github.com/apache/spark/pull/37923#discussion_r976714525
##########
python/pyspark/pandas/groupby.py:
##########
@@ -993,6 +993,105 @@ def nth(self, n: int) -> FrameLike:
return self._prepare_return(DataFrame(internal))
+ def prod(self, numeric_only: Optional[bool] = True, min_count: int = 0) -> FrameLike:
+ """
+ Compute prod of groups.
+
+ .. versionadded:: 3.4.0
+
+ Parameters
+ ----------
+ numeric_only : bool, default False
+ Include only float, int, boolean columns. If None, will attempt to use
+ everything, then use only numeric data.
+
+ min_count: int, default 0
+ The required number of valid values to perform the operation.
+ If fewer than min_count non-NA values are present the result will be NA.
+
+ Returns
+ -------
+ Series or DataFrame
+
+ See Also
+ --------
+ pyspark.pandas.Series.groupby
+ pyspark.pandas.DataFrame.groupby
+
+ Examples
+ --------
+ >>> df = ps.DataFrame({'A': [1, 1, 2, 1, 2],
+ ... 'B': [np.nan, 2, 3, 4, 5],
+ ... 'C': [1, 2, 1, 1, 2],
+ ... 'D': [True, False, True, False, True]})
+
+ Groupby one column and return the prod of the remaining columns in
+ each group.
+
+ >>> df.groupby('A').prod().sort_index()
+ B C D
+ A
+ 1 8.0 2 0
+ 2 15.0 2 1
+
+ >>> df.groupby('A').prod(min_count=3).sort_index()
+ B C D
+ A
+ 1 NaN 2.0 0.0
+ 2 NaN NaN NaN
+ """
+
+ self._validate_agg_columns(numeric_only=numeric_only, function_name="prod")
+
+ groupkey_names = [SPARK_INDEX_NAME_FORMAT(i) for i in range(len(self._groupkeys))]
+ internal, agg_columns, sdf = self._prepare_reduce(
+ groupkey_names=groupkey_names,
+ accepted_spark_types=(NumericType, BooleanType),
+ bool_to_numeric=True,
+ )
+
+ psdf: DataFrame = DataFrame(internal)
+ if len(psdf._internal.column_labels) > 0:
+
+ stat_exprs = []
+ for label in psdf._internal.column_labels:
+ tmp_count_column = verify_temp_column_name(sdf, "__tmp_%s_count_col__" % label[0])
+ psser = psdf._psser_for(label)
+ column = psser._dtype_op.nan_to_null(psser).spark.column
+ data_type = psser.spark.data_type
+
+ if isinstance(data_type, IntegralType):
+ stat_exprs.append(F.product(column).cast(data_type).alias(label[0]))
+ else:
+ stat_exprs.append(F.product(column).alias(label[0]))
Review Comment:
It makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #37923: [SPARK-40334][PS] Implement `GroupBy.prod`
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #37923:
URL: https://github.com/apache/spark/pull/37923#issuecomment-1250505726
cc @zhengruifeng @xinrong-meng @itholic FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #37923: [SPARK-40334][PS] Implement `GroupBy.prod`
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on PR #37923:
URL: https://github.com/apache/spark/pull/37923#issuecomment-1251698604
```
Oh no! 💥 💔 💥
2 files would be reformatted, 352 files would be left unchanged.
Please run 'dev/reformat-python' script.
1
Error: Process completed with exit code 1.
```
`lint-python` failed, you would install related packages in your env via `pip install -r dev/requirements.txt `, and then reformat the python files via `dev/reformat-python` and check the lint locally via `dev/lint-python`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a diff in pull request #37923: [SPARK-40334][PS] Implement `GroupBy.prod`
Posted by GitBox <gi...@apache.org>.
Yikun commented on code in PR #37923:
URL: https://github.com/apache/spark/pull/37923#discussion_r977131978
##########
python/pyspark/pandas/groupby.py:
##########
@@ -18,7 +18,6 @@
"""
A wrapper for GroupedData to behave similar to pandas GroupBy.
"""
-
Review Comment:
Friendly note, looks like it is not recovered yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin commented on a diff in pull request #37923: [SPARK-40334][PS] Implement `GroupBy.prod`
Posted by GitBox <gi...@apache.org>.
ayudovin commented on code in PR #37923:
URL: https://github.com/apache/spark/pull/37923#discussion_r974514017
##########
python/pyspark/pandas/groupby.py:
##########
@@ -993,6 +993,98 @@ def nth(self, n: int) -> FrameLike:
return self._prepare_return(DataFrame(internal))
+ def prod(self, numeric_only: Optional[bool] = True, min_count: int = 0):
+ """
+ Compute prod of groups.
+
+ Parameters
+ ----------
+ numeric_only : bool, default False
+ Include only float, int, boolean columns. If None, will attempt to use
+ everything, then use only numeric data.
+
+ min_count: int, default 0
+ The required number of valid values to perform the operation.
+ If fewer than min_count non-NA values are present the result will be NA.
+
+ .. versionadded:: 3.4.0
+
+ Returns
+ -------
+ pyspark.pandas.Series or pyspark.pandas.DataFrame
+
+ See Also
+ --------
+ pyspark.pandas.Series.groupby
+ pyspark.pandas.DataFrame.groupby
+
+ Examples
+ --------
+ >>> df = ps.DataFrame({'A': [1, 1, 2, 1, 2],
+ ... 'B': [np.nan, 2, 3, 4, 5],
+ ... 'C': [1, 2, 1, 1, 2],
+ ... 'D': [True, False, True, False, True]})
+
+ Groupby one column and return the prod of the remaining columns in
+ each group.
+
+ >>> df.groupby('A').prod().sort_index()
+ B C D
+ A
+ 1 8.0 2 0
+ 2 15.0 2 11
+
+ >>> df.groupby('A').prod(min_count=3).sort_index()
+ B C D
+ A
+ 1 NaN 2 0
+ 2 NaN NaN NaN
+ """
+
+ self._validate_agg_columns(numeric_only=numeric_only, function_name="prod")
+
+ groupkey_names = [SPARK_INDEX_NAME_FORMAT(i) for i in range(len(self._groupkeys))]
+ internal, agg_columns, sdf = self._prepare_reduce(
+ groupkey_names=groupkey_names,
+ accepted_spark_types=(NumericType, BooleanType),
+ bool_to_numeric=True,
+ )
+
+ psdf: DataFrame = DataFrame(internal)
+ if len(psdf._internal.column_labels) > 0:
+
+ stat_exprs = []
+ for label in psdf._internal.column_labels:
+ psser = psdf._psser_for(label)
+ column = psser._dtype_op.nan_to_null(psser).spark.column
+ data_type = psser.spark.data_type
+
+ if isinstance(data_type, IntegralType):
+ stat_exprs.append(F.product(column).cast(data_type).alias(f"{label[0]}"))
+ else:
+ stat_exprs.append(F.product(column).alias(f"{label[0]}"))
+
+ stat_exprs.append(F.count(column).alias(f"{label[0]}_count"))
Review Comment:
yes, you're right
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #37923: [SPARK-40334][PS] Implement `GroupBy.prod`
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on code in PR #37923:
URL: https://github.com/apache/spark/pull/37923#discussion_r977126652
##########
python/pyspark/pandas/groupby.py:
##########
@@ -993,6 +993,115 @@ def nth(self, n: int) -> FrameLike:
return self._prepare_return(DataFrame(internal))
+ def prod(self, numeric_only: Optional[bool] = True, min_count: int = 0) -> FrameLike:
+ """
+ Compute prod of groups.
+
+ .. versionadded:: 3.4.0
+
+ Parameters
+ ----------
+ numeric_only : bool, default False
+ Include only float, int, boolean columns. If None, will attempt to use
+ everything, then use only numeric data.
+
+ min_count: int, default 0
+ The required number of valid values to perform the operation.
+ If fewer than min_count non-NA values are present the result will be NA.
+
+ Returns
+ -------
+ Series or DataFrame
+ Computed prod of values within each group.
+
+ See Also
+ --------
+ pyspark.pandas.Series.groupby
+ pyspark.pandas.DataFrame.groupby
+
+ Examples
+ --------
+ >>> df = ps.DataFrame(
+ ... {
+ ... "A": [1, 1, 2, 1, 2],
+ ... "B": [np.nan, 2, 3, 4, 5],
+ ... "C": [1, 2, 1, 1, 2],
+ ... "D": [True, False, True, False, True],
+ ... }
+ ... )
+
+ Groupby one column and return the prod of the remaining columns in
+ each group.
+
+ >>> df.groupby('A').prod().sort_index()
+ B C D
+ A
+ 1 8.0 2 0
+ 2 15.0 2 1
+
+ >>> df.groupby('A').prod(min_count=3).sort_index()
+ B C D
+ A
+ 1 NaN 2.0 0.0
+ 2 NaN NaN NaN
+ """
+
+ self._validate_agg_columns(numeric_only=numeric_only, function_name="prod")
+
+ groupkey_names = [SPARK_INDEX_NAME_FORMAT(i) for i in range(len(self._groupkeys))]
+ internal, agg_columns, sdf = self._prepare_reduce(
+ groupkey_names=groupkey_names,
+ accepted_spark_types=(NumericType, BooleanType),
+ bool_to_numeric=True,
+ )
+
+ psdf: DataFrame = DataFrame(internal)
+ if len(psdf._internal.column_labels) > 0:
+
+ stat_exprs = []
+ for label in psdf._internal.column_labels:
+ label_name = label[0]
+ tmp_count_column_name = verify_temp_column_name(
+ sdf, "__tmp_%s_count_col__" % label_name
+ )
+ psser = psdf._psser_for(label)
+ column = psser._dtype_op.nan_to_null(psser).spark.column
+ data_type = psser.spark.data_type
+
+ if isinstance(data_type, IntegralType):
Review Comment:
for `IntegralType `, maybe always returns `LongType`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #37923: [SPARK-40334][PS] Implement `GroupBy.prod`
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on PR #37923:
URL: https://github.com/apache/spark/pull/37923#issuecomment-1257315237
Merged into master, thank you @ayudovin for working on it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #37923: [SPARK-40334][PS] Implement `GroupBy.prod`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on PR #37923:
URL: https://github.com/apache/spark/pull/37923#issuecomment-1251551212
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin commented on a diff in pull request #37923: [SPARK-40334][PS] Implement `GroupBy.prod`
Posted by GitBox <gi...@apache.org>.
ayudovin commented on code in PR #37923:
URL: https://github.com/apache/spark/pull/37923#discussion_r977573852
##########
python/pyspark/pandas/groupby.py:
##########
@@ -993,6 +993,115 @@ def nth(self, n: int) -> FrameLike:
return self._prepare_return(DataFrame(internal))
+ def prod(self, numeric_only: Optional[bool] = True, min_count: int = 0) -> FrameLike:
+ """
+ Compute prod of groups.
+
+ .. versionadded:: 3.4.0
+
+ Parameters
+ ----------
+ numeric_only : bool, default False
+ Include only float, int, boolean columns. If None, will attempt to use
+ everything, then use only numeric data.
+
+ min_count: int, default 0
+ The required number of valid values to perform the operation.
+ If fewer than min_count non-NA values are present the result will be NA.
+
+ Returns
+ -------
+ Series or DataFrame
+ Computed prod of values within each group.
+
+ See Also
+ --------
+ pyspark.pandas.Series.groupby
+ pyspark.pandas.DataFrame.groupby
+
+ Examples
+ --------
+ >>> df = ps.DataFrame(
+ ... {
+ ... "A": [1, 1, 2, 1, 2],
+ ... "B": [np.nan, 2, 3, 4, 5],
+ ... "C": [1, 2, 1, 1, 2],
+ ... "D": [True, False, True, False, True],
+ ... }
+ ... )
+
+ Groupby one column and return the prod of the remaining columns in
+ each group.
+
+ >>> df.groupby('A').prod().sort_index()
+ B C D
+ A
+ 1 8.0 2 0
+ 2 15.0 2 1
+
+ >>> df.groupby('A').prod(min_count=3).sort_index()
+ B C D
+ A
+ 1 NaN 2.0 0.0
+ 2 NaN NaN NaN
+ """
+
+ self._validate_agg_columns(numeric_only=numeric_only, function_name="prod")
+
+ groupkey_names = [SPARK_INDEX_NAME_FORMAT(i) for i in range(len(self._groupkeys))]
+ internal, agg_columns, sdf = self._prepare_reduce(
+ groupkey_names=groupkey_names,
+ accepted_spark_types=(NumericType, BooleanType),
+ bool_to_numeric=True,
+ )
+
+ psdf: DataFrame = DataFrame(internal)
+ if len(psdf._internal.column_labels) > 0:
+
+ stat_exprs = []
+ for label in psdf._internal.column_labels:
+ label_name = label[0]
+ tmp_count_column_name = verify_temp_column_name(
+ sdf, "__tmp_%s_count_col__" % label_name
+ )
+ psser = psdf._psser_for(label)
+ column = psser._dtype_op.nan_to_null(psser).spark.column
+ data_type = psser.spark.data_type
+
+ if isinstance(data_type, IntegralType):
+ stat_exprs.append(F.product(column).cast(data_type).alias(label_name))
+ else:
+ stat_exprs.append(F.product(column).alias(label_name))
+
+ if min_count > 0:
+ stat_exprs.append(F.count(column).alias(tmp_count_column_name))
+
+ sdf = sdf.groupby(*groupkey_names).agg(*stat_exprs)
Review Comment:
yes, we can, I applied changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #37923: [SPARK-40334][PS] Implement `GroupBy.prod`
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on code in PR #37923:
URL: https://github.com/apache/spark/pull/37923#discussion_r978229729
##########
python/pyspark/pandas/groupby.py:
##########
@@ -3237,10 +3337,10 @@ def _validate_agg_columns(self, numeric_only: Optional[bool], function_name: str
if not numeric_only:
if has_non_numeric:
warnings.warn(
- "Dropping invalid columns in DataFrameGroupBy.mean is deprecated. "
Review Comment:
nice catch!
##########
python/pyspark/pandas/groupby.py:
##########
@@ -993,6 +993,106 @@ def nth(self, n: int) -> FrameLike:
return self._prepare_return(DataFrame(internal))
+ def prod(self, numeric_only: Optional[bool] = True, min_count: int = 0) -> FrameLike:
+ """
+ Compute prod of groups.
+
+ .. versionadded:: 3.4.0
+
+ Parameters
+ ----------
+ numeric_only : bool, default False
+ Include only float, int, boolean columns. If None, will attempt to use
+ everything, then use only numeric data.
+
+ min_count: int, default 0
+ The required number of valid values to perform the operation.
+ If fewer than min_count non-NA values are present the result will be NA.
+
+ Returns
+ -------
+ Series or DataFrame
+ Computed prod of values within each group.
+
+ See Also
+ --------
+ pyspark.pandas.Series.groupby
+ pyspark.pandas.DataFrame.groupby
+
+ Examples
+ --------
+ >>> import numpy as np
+ >>> df = ps.DataFrame(
+ ... {
+ ... "A": [1, 1, 2, 1, 2],
+ ... "B": [np.nan, 2, 3, 4, 5],
+ ... "C": [1, 2, 1, 1, 2],
+ ... "D": [True, False, True, False, True],
+ ... }
+ ... )
+
+ Groupby one column and return the prod of the remaining columns in
+ each group.
+
+ >>> df.groupby('A').prod().sort_index()
+ B C D
+ A
+ 1 8.0 2 0
+ 2 15.0 2 1
+
+ >>> df.groupby('A').prod(min_count=3).sort_index()
+ B C D
+ A
+ 1 NaN 2.0 0.0
+ 2 NaN NaN NaN
+ """
+
+ self._validate_agg_columns(numeric_only=numeric_only, function_name="prod")
+
+ groupkey_names = [SPARK_INDEX_NAME_FORMAT(i) for i in range(len(self._groupkeys))]
+ internal, agg_columns, sdf = self._prepare_reduce(
+ groupkey_names=groupkey_names,
+ accepted_spark_types=(NumericType, BooleanType),
+ bool_to_numeric=True,
+ )
+
+ psdf: DataFrame = DataFrame(internal)
+ if len(psdf._internal.column_labels) > 0:
+
+ stat_exprs = []
+ for label in psdf._internal.column_labels:
+ psser = psdf._psser_for(label)
+ column = psser._dtype_op.nan_to_null(psser).spark.column
+ data_type = psser.spark.data_type
+ aggregating = (
+ F.product(column).cast(data_type)
+ if isinstance(data_type, IntegralType)
+ else F.product(column)
+ )
+
+ if min_count > 0:
+ prod_scol = F.when(F.count(column) < min_count, F.lit(None)).otherwise(
Review Comment:
```suggestion
prod_scol = F.when(F.count(F.when(~F.isnull(column), F.lit(0))) < min_count, F.lit(None)).otherwise(
```
sorry, I forgot to filter out invalid values in previous suggestion
##########
python/pyspark/pandas/groupby.py:
##########
@@ -993,6 +993,106 @@ def nth(self, n: int) -> FrameLike:
return self._prepare_return(DataFrame(internal))
+ def prod(self, numeric_only: Optional[bool] = True, min_count: int = 0) -> FrameLike:
+ """
+ Compute prod of groups.
+
+ .. versionadded:: 3.4.0
+
+ Parameters
+ ----------
+ numeric_only : bool, default False
+ Include only float, int, boolean columns. If None, will attempt to use
+ everything, then use only numeric data.
+
+ min_count: int, default 0
+ The required number of valid values to perform the operation.
+ If fewer than min_count non-NA values are present the result will be NA.
+
+ Returns
+ -------
+ Series or DataFrame
+ Computed prod of values within each group.
+
+ See Also
+ --------
+ pyspark.pandas.Series.groupby
+ pyspark.pandas.DataFrame.groupby
+
+ Examples
+ --------
+ >>> import numpy as np
+ >>> df = ps.DataFrame(
+ ... {
+ ... "A": [1, 1, 2, 1, 2],
+ ... "B": [np.nan, 2, 3, 4, 5],
+ ... "C": [1, 2, 1, 1, 2],
+ ... "D": [True, False, True, False, True],
+ ... }
+ ... )
+
+ Groupby one column and return the prod of the remaining columns in
+ each group.
+
+ >>> df.groupby('A').prod().sort_index()
+ B C D
+ A
+ 1 8.0 2 0
+ 2 15.0 2 1
+
+ >>> df.groupby('A').prod(min_count=3).sort_index()
+ B C D
+ A
+ 1 NaN 2.0 0.0
+ 2 NaN NaN NaN
+ """
+
+ self._validate_agg_columns(numeric_only=numeric_only, function_name="prod")
+
+ groupkey_names = [SPARK_INDEX_NAME_FORMAT(i) for i in range(len(self._groupkeys))]
+ internal, agg_columns, sdf = self._prepare_reduce(
+ groupkey_names=groupkey_names,
+ accepted_spark_types=(NumericType, BooleanType),
+ bool_to_numeric=True,
+ )
+
+ psdf: DataFrame = DataFrame(internal)
+ if len(psdf._internal.column_labels) > 0:
+
+ stat_exprs = []
+ for label in psdf._internal.column_labels:
+ psser = psdf._psser_for(label)
+ column = psser._dtype_op.nan_to_null(psser).spark.column
+ data_type = psser.spark.data_type
+ aggregating = (
+ F.product(column).cast(data_type)
Review Comment:
```suggestion
F.product(column).cast("long")
```
what about always casting to `long`, since the `prod` easily output large numbers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #37923: [SPARK-40334][PS] Implement `GroupBy.prod`
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on code in PR #37923:
URL: https://github.com/apache/spark/pull/37923#discussion_r977124090
##########
python/pyspark/pandas/groupby.py:
##########
@@ -993,6 +993,115 @@ def nth(self, n: int) -> FrameLike:
return self._prepare_return(DataFrame(internal))
+ def prod(self, numeric_only: Optional[bool] = True, min_count: int = 0) -> FrameLike:
+ """
+ Compute prod of groups.
+
+ .. versionadded:: 3.4.0
+
+ Parameters
+ ----------
+ numeric_only : bool, default False
+ Include only float, int, boolean columns. If None, will attempt to use
+ everything, then use only numeric data.
+
+ min_count: int, default 0
+ The required number of valid values to perform the operation.
+ If fewer than min_count non-NA values are present the result will be NA.
+
+ Returns
+ -------
+ Series or DataFrame
+ Computed prod of values within each group.
+
+ See Also
+ --------
+ pyspark.pandas.Series.groupby
+ pyspark.pandas.DataFrame.groupby
+
+ Examples
+ --------
+ >>> df = ps.DataFrame(
+ ... {
+ ... "A": [1, 1, 2, 1, 2],
+ ... "B": [np.nan, 2, 3, 4, 5],
+ ... "C": [1, 2, 1, 1, 2],
+ ... "D": [True, False, True, False, True],
+ ... }
+ ... )
+
+ Groupby one column and return the prod of the remaining columns in
+ each group.
+
+ >>> df.groupby('A').prod().sort_index()
+ B C D
+ A
+ 1 8.0 2 0
+ 2 15.0 2 1
+
+ >>> df.groupby('A').prod(min_count=3).sort_index()
+ B C D
+ A
+ 1 NaN 2.0 0.0
+ 2 NaN NaN NaN
+ """
+
+ self._validate_agg_columns(numeric_only=numeric_only, function_name="prod")
+
+ groupkey_names = [SPARK_INDEX_NAME_FORMAT(i) for i in range(len(self._groupkeys))]
+ internal, agg_columns, sdf = self._prepare_reduce(
+ groupkey_names=groupkey_names,
+ accepted_spark_types=(NumericType, BooleanType),
+ bool_to_numeric=True,
+ )
+
+ psdf: DataFrame = DataFrame(internal)
+ if len(psdf._internal.column_labels) > 0:
+
+ stat_exprs = []
+ for label in psdf._internal.column_labels:
+ label_name = label[0]
+ tmp_count_column_name = verify_temp_column_name(
+ sdf, "__tmp_%s_count_col__" % label_name
+ )
+ psser = psdf._psser_for(label)
+ column = psser._dtype_op.nan_to_null(psser).spark.column
+ data_type = psser.spark.data_type
+
+ if isinstance(data_type, IntegralType):
+ stat_exprs.append(F.product(column).cast(data_type).alias(label_name))
+ else:
+ stat_exprs.append(F.product(column).alias(label_name))
+
+ if min_count > 0:
+ stat_exprs.append(F.count(column).alias(tmp_count_column_name))
+
+ sdf = sdf.groupby(*groupkey_names).agg(*stat_exprs)
Review Comment:
Can we simplify the logic like this?
```suggestion
for label in psdf._internal.column_labels:
psser = psdf._psser_for(label)
scol = psser._dtype_op.nan_to_null(psser).spark.column
prod_scol = F.product(scol)
if min_count > 0:
prod_scol = F.when(F.count(scol) < min_count, F.lit(None)).otherwise(F.product(scol))
else:
prod_scol = F.product(scol)
stat_exprs.append(
prod_scol.alias(psser._internal.data_spark_column_names[0])
)
sdf = sdf.groupby(*groupkey_names).agg(*stat_exprs)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin commented on a diff in pull request #37923: [SPARK-40334][PS] Implement `GroupBy.prod`
Posted by GitBox <gi...@apache.org>.
ayudovin commented on code in PR #37923:
URL: https://github.com/apache/spark/pull/37923#discussion_r977578297
##########
python/pyspark/pandas/groupby.py:
##########
@@ -993,6 +993,115 @@ def nth(self, n: int) -> FrameLike:
return self._prepare_return(DataFrame(internal))
+ def prod(self, numeric_only: Optional[bool] = True, min_count: int = 0) -> FrameLike:
+ """
+ Compute prod of groups.
+
+ .. versionadded:: 3.4.0
+
+ Parameters
+ ----------
+ numeric_only : bool, default False
+ Include only float, int, boolean columns. If None, will attempt to use
+ everything, then use only numeric data.
+
+ min_count: int, default 0
+ The required number of valid values to perform the operation.
+ If fewer than min_count non-NA values are present the result will be NA.
+
+ Returns
+ -------
+ Series or DataFrame
+ Computed prod of values within each group.
+
+ See Also
+ --------
+ pyspark.pandas.Series.groupby
+ pyspark.pandas.DataFrame.groupby
+
+ Examples
+ --------
+ >>> df = ps.DataFrame(
+ ... {
+ ... "A": [1, 1, 2, 1, 2],
+ ... "B": [np.nan, 2, 3, 4, 5],
+ ... "C": [1, 2, 1, 1, 2],
+ ... "D": [True, False, True, False, True],
+ ... }
+ ... )
+
+ Groupby one column and return the prod of the remaining columns in
+ each group.
+
+ >>> df.groupby('A').prod().sort_index()
+ B C D
+ A
+ 1 8.0 2 0
+ 2 15.0 2 1
+
+ >>> df.groupby('A').prod(min_count=3).sort_index()
+ B C D
+ A
+ 1 NaN 2.0 0.0
+ 2 NaN NaN NaN
+ """
+
+ self._validate_agg_columns(numeric_only=numeric_only, function_name="prod")
+
+ groupkey_names = [SPARK_INDEX_NAME_FORMAT(i) for i in range(len(self._groupkeys))]
+ internal, agg_columns, sdf = self._prepare_reduce(
+ groupkey_names=groupkey_names,
+ accepted_spark_types=(NumericType, BooleanType),
+ bool_to_numeric=True,
+ )
+
+ psdf: DataFrame = DataFrame(internal)
+ if len(psdf._internal.column_labels) > 0:
+
+ stat_exprs = []
+ for label in psdf._internal.column_labels:
+ label_name = label[0]
+ tmp_count_column_name = verify_temp_column_name(
+ sdf, "__tmp_%s_count_col__" % label_name
+ )
+ psser = psdf._psser_for(label)
+ column = psser._dtype_op.nan_to_null(psser).spark.column
+ data_type = psser.spark.data_type
+
+ if isinstance(data_type, IntegralType):
Review Comment:
Do you mean we won't have a case when data_type is not the LongType?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #37923: [SPARK-40334][PS] Implement `GroupBy.prod`
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on PR #37923:
URL: https://github.com/apache/spark/pull/37923#issuecomment-1250593684
also cc @Yikun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a diff in pull request #37923: [SPARK-40334][PS] Implement `GroupBy.prod`
Posted by GitBox <gi...@apache.org>.
Yikun commented on code in PR #37923:
URL: https://github.com/apache/spark/pull/37923#discussion_r973821134
##########
python/pyspark/pandas/groupby.py:
##########
@@ -993,6 +993,98 @@ def nth(self, n: int) -> FrameLike:
return self._prepare_return(DataFrame(internal))
+ def prod(self, numeric_only: Optional[bool] = True, min_count: int = 0):
+ """
+ Compute prod of groups.
+
+ Parameters
+ ----------
+ numeric_only : bool, default False
+ Include only float, int, boolean columns. If None, will attempt to use
+ everything, then use only numeric data.
+
+ min_count: int, default 0
+ The required number of valid values to perform the operation.
+ If fewer than min_count non-NA values are present the result will be NA.
+
+ .. versionadded:: 3.4.0
+
+ Returns
+ -------
+ pyspark.pandas.Series or pyspark.pandas.DataFrame
+
+ See Also
+ --------
+ pyspark.pandas.Series.groupby
+ pyspark.pandas.DataFrame.groupby
+
+ Examples
+ --------
+ >>> df = ps.DataFrame({'A': [1, 1, 2, 1, 2],
+ ... 'B': [np.nan, 2, 3, 4, 5],
+ ... 'C': [1, 2, 1, 1, 2],
+ ... 'D': [True, False, True, False, True]})
+
+ Groupby one column and return the prod of the remaining columns in
+ each group.
+
+ >>> df.groupby('A').prod().sort_index()
+ B C D
+ A
+ 1 8.0 2 0
+ 2 15.0 2 11
+
+ >>> df.groupby('A').prod(min_count=3).sort_index()
+ B C D
+ A
+ 1 NaN 2 0
Review Comment:
```suggestion
1 NaN 2.0 0.0
```
##########
python/pyspark/pandas/groupby.py:
##########
@@ -61,7 +61,7 @@
NumericType,
StructField,
StructType,
- StringType,
+ StringType, IntegralType,
Review Comment:
```suggestion
StringType,
IntegralType,
```
##########
python/pyspark/pandas/groupby.py:
##########
@@ -993,6 +993,98 @@ def nth(self, n: int) -> FrameLike:
return self._prepare_return(DataFrame(internal))
+ def prod(self, numeric_only: Optional[bool] = True, min_count: int = 0):
+ """
+ Compute prod of groups.
+
+ Parameters
+ ----------
+ numeric_only : bool, default False
+ Include only float, int, boolean columns. If None, will attempt to use
+ everything, then use only numeric data.
+
+ min_count: int, default 0
+ The required number of valid values to perform the operation.
+ If fewer than min_count non-NA values are present the result will be NA.
+
+ .. versionadded:: 3.4.0
+
+ Returns
+ -------
+ pyspark.pandas.Series or pyspark.pandas.DataFrame
+
+ See Also
+ --------
+ pyspark.pandas.Series.groupby
+ pyspark.pandas.DataFrame.groupby
+
+ Examples
+ --------
+ >>> df = ps.DataFrame({'A': [1, 1, 2, 1, 2],
+ ... 'B': [np.nan, 2, 3, 4, 5],
+ ... 'C': [1, 2, 1, 1, 2],
+ ... 'D': [True, False, True, False, True]})
+
+ Groupby one column and return the prod of the remaining columns in
+ each group.
+
+ >>> df.groupby('A').prod().sort_index()
+ B C D
+ A
+ 1 8.0 2 0
+ 2 15.0 2 11
+
+ >>> df.groupby('A').prod(min_count=3).sort_index()
+ B C D
+ A
+ 1 NaN 2 0
+ 2 NaN NaN NaN
+ """
+
+ self._validate_agg_columns(numeric_only=numeric_only, function_name="prod")
+
+ groupkey_names = [SPARK_INDEX_NAME_FORMAT(i) for i in range(len(self._groupkeys))]
+ internal, agg_columns, sdf = self._prepare_reduce(
+ groupkey_names=groupkey_names,
+ accepted_spark_types=(NumericType, BooleanType),
+ bool_to_numeric=True,
+ )
+
+ psdf: DataFrame = DataFrame(internal)
+ if len(psdf._internal.column_labels) > 0:
+
+ stat_exprs = []
+ for label in psdf._internal.column_labels:
+ psser = psdf._psser_for(label)
+ column = psser._dtype_op.nan_to_null(psser).spark.column
+ data_type = psser.spark.data_type
+
+ if isinstance(data_type, IntegralType):
+ stat_exprs.append(F.product(column).cast(data_type).alias(f"{label[0]}"))
+ else:
+ stat_exprs.append(F.product(column).alias(f"{label[0]}"))
+
+ stat_exprs.append(F.count(column).alias(f"{label[0]}_count"))
Review Comment:
Looks like we also don't need this helper column if min_count==0, right?
##########
python/pyspark/pandas/groupby.py:
##########
@@ -993,6 +993,98 @@ def nth(self, n: int) -> FrameLike:
return self._prepare_return(DataFrame(internal))
+ def prod(self, numeric_only: Optional[bool] = True, min_count: int = 0):
+ """
+ Compute prod of groups.
+
+ Parameters
+ ----------
+ numeric_only : bool, default False
+ Include only float, int, boolean columns. If None, will attempt to use
+ everything, then use only numeric data.
+
+ min_count: int, default 0
+ The required number of valid values to perform the operation.
+ If fewer than min_count non-NA values are present the result will be NA.
+
+ .. versionadded:: 3.4.0
+
+ Returns
+ -------
+ pyspark.pandas.Series or pyspark.pandas.DataFrame
+
+ See Also
+ --------
+ pyspark.pandas.Series.groupby
+ pyspark.pandas.DataFrame.groupby
+
+ Examples
+ --------
+ >>> df = ps.DataFrame({'A': [1, 1, 2, 1, 2],
+ ... 'B': [np.nan, 2, 3, 4, 5],
+ ... 'C': [1, 2, 1, 1, 2],
+ ... 'D': [True, False, True, False, True]})
+
+ Groupby one column and return the prod of the remaining columns in
+ each group.
+
+ >>> df.groupby('A').prod().sort_index()
+ B C D
+ A
+ 1 8.0 2 0
+ 2 15.0 2 11
Review Comment:
```suggestion
2 15.0 2 1
```
##########
python/pyspark/pandas/groupby.py:
##########
@@ -993,6 +993,98 @@ def nth(self, n: int) -> FrameLike:
return self._prepare_return(DataFrame(internal))
+ def prod(self, numeric_only: Optional[bool] = True, min_count: int = 0):
+ """
+ Compute prod of groups.
+
+ Parameters
+ ----------
+ numeric_only : bool, default False
+ Include only float, int, boolean columns. If None, will attempt to use
+ everything, then use only numeric data.
Review Comment:
```suggestion
numeric_only : bool, default False
Include only float, int, boolean columns. If None, will attempt to use
everything, then use only numeric data.
```
##########
python/pyspark/pandas/groupby.py:
##########
@@ -993,6 +993,98 @@ def nth(self, n: int) -> FrameLike:
return self._prepare_return(DataFrame(internal))
+ def prod(self, numeric_only: Optional[bool] = True, min_count: int = 0):
+ """
+ Compute prod of groups.
+
+ Parameters
+ ----------
+ numeric_only : bool, default False
+ Include only float, int, boolean columns. If None, will attempt to use
+ everything, then use only numeric data.
+
+ min_count: int, default 0
+ The required number of valid values to perform the operation.
+ If fewer than min_count non-NA values are present the result will be NA.
Review Comment:
```suggestion
min_count: int, default 0
The required number of valid values to perform the operation.
If fewer than min_count non-NA values are present the result will be NA.
```
##########
python/pyspark/pandas/groupby.py:
##########
@@ -993,6 +993,98 @@ def nth(self, n: int) -> FrameLike:
return self._prepare_return(DataFrame(internal))
+ def prod(self, numeric_only: Optional[bool] = True, min_count: int = 0):
+ """
+ Compute prod of groups.
+
+ Parameters
+ ----------
+ numeric_only : bool, default False
+ Include only float, int, boolean columns. If None, will attempt to use
+ everything, then use only numeric data.
+
+ min_count: int, default 0
+ The required number of valid values to perform the operation.
+ If fewer than min_count non-NA values are present the result will be NA.
+
+ .. versionadded:: 3.4.0
Review Comment:
You might want to move this before `Parameters`
##########
python/pyspark/pandas/groupby.py:
##########
@@ -993,6 +993,98 @@ def nth(self, n: int) -> FrameLike:
return self._prepare_return(DataFrame(internal))
+ def prod(self, numeric_only: Optional[bool] = True, min_count: int = 0):
+ """
+ Compute prod of groups.
+
+ Parameters
+ ----------
+ numeric_only : bool, default False
+ Include only float, int, boolean columns. If None, will attempt to use
+ everything, then use only numeric data.
+
+ min_count: int, default 0
+ The required number of valid values to perform the operation.
+ If fewer than min_count non-NA values are present the result will be NA.
+
+ .. versionadded:: 3.4.0
+
+ Returns
+ -------
+ pyspark.pandas.Series or pyspark.pandas.DataFrame
+
+ See Also
+ --------
+ pyspark.pandas.Series.groupby
+ pyspark.pandas.DataFrame.groupby
+
+ Examples
+ --------
+ >>> df = ps.DataFrame({'A': [1, 1, 2, 1, 2],
+ ... 'B': [np.nan, 2, 3, 4, 5],
+ ... 'C': [1, 2, 1, 1, 2],
+ ... 'D': [True, False, True, False, True]})
+
+ Groupby one column and return the prod of the remaining columns in
+ each group.
+
+ >>> df.groupby('A').prod().sort_index()
+ B C D
+ A
+ 1 8.0 2 0
+ 2 15.0 2 11
+
+ >>> df.groupby('A').prod(min_count=3).sort_index()
+ B C D
+ A
+ 1 NaN 2 0
+ 2 NaN NaN NaN
+ """
+
+ self._validate_agg_columns(numeric_only=numeric_only, function_name="prod")
+
+ groupkey_names = [SPARK_INDEX_NAME_FORMAT(i) for i in range(len(self._groupkeys))]
+ internal, agg_columns, sdf = self._prepare_reduce(
+ groupkey_names=groupkey_names,
+ accepted_spark_types=(NumericType, BooleanType),
+ bool_to_numeric=True,
+ )
+
+ psdf: DataFrame = DataFrame(internal)
+ if len(psdf._internal.column_labels) > 0:
+
+ stat_exprs = []
+ for label in psdf._internal.column_labels:
+ psser = psdf._psser_for(label)
+ column = psser._dtype_op.nan_to_null(psser).spark.column
+ data_type = psser.spark.data_type
+
+ if isinstance(data_type, IntegralType):
+ stat_exprs.append(F.product(column).cast(data_type).alias(f"{label[0]}"))
+ else:
+ stat_exprs.append(F.product(column).alias(f"{label[0]}"))
+
+ stat_exprs.append(F.count(column).alias(f"{label[0]}_count"))
+
+ sdf = sdf.groupby(*groupkey_names).agg(*stat_exprs)
+
+ if min_count > 0:
+ for label in psdf._internal.column_labels:
+ sdf = sdf.withColumn(f"{label[0]}", F.when(F.col(f"{label[0]}_count").__ge__(min_count),
Review Comment:
we usually choose to create a temporary column with a special name and verify (`verify_temp_column_name`), and then drop it.
see below as reference:
https://github.com/apache/spark/blob/master/python/pyspark/pandas/groupby.py#L1024-L1032
##########
python/pyspark/pandas/groupby.py:
##########
@@ -993,6 +993,98 @@ def nth(self, n: int) -> FrameLike:
return self._prepare_return(DataFrame(internal))
+ def prod(self, numeric_only: Optional[bool] = True, min_count: int = 0):
+ """
+ Compute prod of groups.
+
+ Parameters
+ ----------
+ numeric_only : bool, default False
+ Include only float, int, boolean columns. If None, will attempt to use
+ everything, then use only numeric data.
+
+ min_count: int, default 0
+ The required number of valid values to perform the operation.
+ If fewer than min_count non-NA values are present the result will be NA.
+
+ .. versionadded:: 3.4.0
+
+ Returns
+ -------
+ pyspark.pandas.Series or pyspark.pandas.DataFrame
+
+ See Also
+ --------
+ pyspark.pandas.Series.groupby
+ pyspark.pandas.DataFrame.groupby
+
+ Examples
+ --------
+ >>> df = ps.DataFrame({'A': [1, 1, 2, 1, 2],
+ ... 'B': [np.nan, 2, 3, 4, 5],
+ ... 'C': [1, 2, 1, 1, 2],
+ ... 'D': [True, False, True, False, True]})
+
+ Groupby one column and return the prod of the remaining columns in
+ each group.
+
+ >>> df.groupby('A').prod().sort_index()
+ B C D
+ A
+ 1 8.0 2 0
+ 2 15.0 2 11
+
+ >>> df.groupby('A').prod(min_count=3).sort_index()
+ B C D
+ A
+ 1 NaN 2 0
+ 2 NaN NaN NaN
+ """
+
+ self._validate_agg_columns(numeric_only=numeric_only, function_name="prod")
+
+ groupkey_names = [SPARK_INDEX_NAME_FORMAT(i) for i in range(len(self._groupkeys))]
+ internal, agg_columns, sdf = self._prepare_reduce(
+ groupkey_names=groupkey_names,
+ accepted_spark_types=(NumericType, BooleanType),
+ bool_to_numeric=True,
+ )
+
+ psdf: DataFrame = DataFrame(internal)
+ if len(psdf._internal.column_labels) > 0:
+
+ stat_exprs = []
+ for label in psdf._internal.column_labels:
+ psser = psdf._psser_for(label)
+ column = psser._dtype_op.nan_to_null(psser).spark.column
+ data_type = psser.spark.data_type
+
+ if isinstance(data_type, IntegralType):
+ stat_exprs.append(F.product(column).cast(data_type).alias(f"{label[0]}"))
Review Comment:
nit:
> let's don't use f-format. we dropped Python 3.5 so it's technically fine but I think it's better to don't break that support only because of this one string format.
https://github.com/apache/spark/pull/36509#discussion_r874263076
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #37923: [SPARK-40334][PS] Implement `GroupBy.prod`
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on code in PR #37923:
URL: https://github.com/apache/spark/pull/37923#discussion_r974779483
##########
python/pyspark/pandas/groupby.py:
##########
@@ -44,6 +43,7 @@
)
import warnings
+import numpy as np
Review Comment:
`numpy` in the docstring was imported in `def _test()`, so no need to import it here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org