You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/09/26 03:13:43 UTC
[GitHub] [spark] zhengruifeng opened a new pull request, #37992: [SPARK-40554][PS] Make `ddof` in `DataFrame.sem` and `Series.sem` accept arbitary integers
zhengruifeng opened a new pull request, #37992:
URL: https://github.com/apache/spark/pull/37992
### What changes were proposed in this pull request?
Make `ddof` in `DataFrame.sem` and `Series.sem` accept arbitary integers
### Why are the changes needed?
for API coverage
### Does this PR introduce _any_ user-facing change?
before:
```
In [1]: import pyspark.pandas as ps
In [2]: psdf = ps.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]})
In [3]: psdf.sem(ddof=2)
---------------------------------------------------------------------------
AssertionError Traceback (most recent call last)
Cell In [3], line 1
----> 1 psdf.sem(ddof=2)
File ~/Dev/spark/python/pyspark/pandas/generic.py:2190, in Frame.sem(self, axis, skipna, ddof, numeric_only)
2120 def sem(
2121 self,
2122 axis: Optional[Axis] = None,
(...)
2125 numeric_only: bool = None,
2126 ) -> Union[Scalar, "Series"]:
2127 """
2128 Return unbiased standard error of the mean over requested axis.
2129
(...)
2188 0.47140452079103173
2189 """
-> 2190 assert ddof in (0, 1)
2192 axis = validate_axis(axis)
2194 if numeric_only is None and axis == 0:
AssertionError:
```
after:
```
In [5]: psdf.sem(ddof=2)
Out[5]:
a 0.816497
b 0.816497
dtype: float64
In [6]: psdf.to_pandas().sem(ddof=2)
/Users/ruifeng.zheng/Dev/spark/python/pyspark/pandas/utils.py:975: PandasAPIOnSparkAdviceWarning: `to_pandas` loads all data into the driver's memory. It should only be used if the resulting pandas DataFrame is expected to be small.
warnings.warn(message, PandasAPIOnSparkAdviceWarning)
Out[6]:
a 0.816497
b 0.816497
dtype: float64
```
### How was this patch tested?
added UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #37992: [SPARK-40554][PS] Make `ddof` in `DataFrame.sem` and `Series.sem` accept arbitary integers
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on PR #37992:
URL: https://github.com/apache/spark/pull/37992#issuecomment-1257591124
Merged into master, thanks @HyukjinKwon for reviews
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng closed pull request #37992: [SPARK-40554][PS] Make `ddof` in `DataFrame.sem` and `Series.sem` accept arbitary integers
Posted by GitBox <gi...@apache.org>.
zhengruifeng closed pull request #37992: [SPARK-40554][PS] Make `ddof` in `DataFrame.sem` and `Series.sem` accept arbitary integers
URL: https://github.com/apache/spark/pull/37992
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org