You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/08/05 04:44:46 UTC
[GitHub] [spark] itholic opened a new pull request #33646: [SPARK-36388][PYTHON] Fix DataFrame groupby-rolling to follow pandas 1.3
itholic opened a new pull request #33646:
URL: https://github.com/apache/spark/pull/33646
### What changes were proposed in this pull request?
This PR proposes to fix `GroupByRolling` to follow latest pandas behavior.
`GroupByRolling` no longer returns grouped-by column in values from pandas 1.3.
Before:
```python
>>> df = pd.DataFrame({"A": [1, 1, 2, 3], "B": [0, 1, 2, 3]})
>>> df.groupby("A").rolling(2).sum()
A B
A
1 0 NaN NaN
1 2.0 1.0
2 2 NaN NaN
3 3 NaN NaN
```
After:
```python
>>> df = pd.DataFrame({"A": [1, 1, 2, 3], "B": [0, 1, 2, 3]})
>>> df.groupby("A").rolling(2).sum()
B
A
1 0 NaN
1 1.0
2 2 NaN
3 3 NaN
```
### Why are the changes needed?
We should follow the behavior of pandas as much as possible.
### Does this PR introduce _any_ user-facing change?
Yes, the result of `GroupByRolling` is changed as described above.
### How was this patch tested?
Unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-894953783
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142207/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33646: [SPARK-36388][PYTHON] Fix DataFrame groupby-rolling to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-893213047
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46576/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-894971483
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46719/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-894936794
**[Test build #142207 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142207/testReport)** for PR 33646 at commit [`a5c169a`](https://github.com/apache/spark/commit/a5c169a6dc3ca4ecdadae0beabac8565def7a4f8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33646: [SPARK-36388][PYTHON] Fix DataFrame groupby-rolling to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-893163413
**[Test build #142066 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142066/testReport)** for PR 33646 at commit [`de59b6e`](https://github.com/apache/spark/commit/de59b6e8b293cfeb3b352ea05cc285a2b4e315fb).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-893964197
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142121/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-894892998
**[Test build #142201 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142201/testReport)** for PR 33646 at commit [`11ea4a2`](https://github.com/apache/spark/commit/11ea4a241e5ab5c6f9ecdcbc4d6eb041712e6be7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33646: [SPARK-36388][PYTHON] Fix DataFrame groupby-rolling to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-893184880
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142066/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #33646:
URL: https://github.com/apache/spark/pull/33646#discussion_r685436498
##########
File path: python/pyspark/pandas/window.py
##########
@@ -706,10 +706,16 @@ def _apply_as_series_or_frame(self, func: Callable[[Column], Column]) -> FrameLi
if groupby._agg_columns_selected:
agg_columns = groupby._agg_columns
else:
+ # pandas doesn't keep the groupkey as a column from 1.3 for DataFrameGroupBy
+ if isinstance(groupby, DataFrameGroupBy):
+ for groupkey in groupby._groupkeys: # type: ignore
+ groupby._column_labels_to_exclude.add( # type: ignore
Review comment:
Shall we not update `groupby._column_labels_to_exclude` directly?
```py
column_labels_to_exclude = groupby._column_labels_to_exclude.copy()
if isinstance(groupby, DataFrameGroupBy):
for ...
column_labels_to_exclude.add( ... )
agg_columns = [ ... if label not in column_labels_to_exclude]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-894022973
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46645/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-894907498
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-894892998
**[Test build #142201 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142201/testReport)** for PR 33646 at commit [`11ea4a2`](https://github.com/apache/spark/commit/11ea4a241e5ab5c6f9ecdcbc4d6eb041712e6be7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-895652387
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142246/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-895633763
**[Test build #142246 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142246/testReport)** for PR 33646 at commit [`c518cdd`](https://github.com/apache/spark/commit/c518cddb7464fbc1a6677469b8df345760a2288b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-895647257
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46753/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-894004368
**[Test build #142133 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142133/testReport)** for PR 33646 at commit [`4b1485e`](https://github.com/apache/spark/commit/4b1485e903f33e94feb349feaa705d52575bd729).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33646: [SPARK-36388][PYTHON] Fix DataFrame groupby-rolling to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-893184880
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-894077638
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142120/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33646: [SPARK-36388][PYTHON] Fix DataFrame groupby-rolling to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-893182196
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46576/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33646: [SPARK-36388][PYTHON] Fix DataFrame groupby-rolling to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-893184880
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142066/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33646: [SPARK-36388][PYTHON] Fix DataFrame groupby-rolling to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-893163413
**[Test build #142066 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142066/testReport)** for PR 33646 at commit [`de59b6e`](https://github.com/apache/spark/commit/de59b6e8b293cfeb3b352ea05cc285a2b4e315fb).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-894955039
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46719/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-894077638
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142120/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-893966214
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46631/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-893981752
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46634/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-893948868
**[Test build #142121 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142121/testReport)** for PR 33646 at commit [`eb464f5`](https://github.com/apache/spark/commit/eb464f57f902b425609a7e7e4649adc07bc0ad9d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33646: [SPARK-36388][PYTHON] Fix DataFrame groupby-rolling to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-893163413
**[Test build #142066 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142066/testReport)** for PR 33646 at commit [`de59b6e`](https://github.com/apache/spark/commit/de59b6e8b293cfeb3b352ea05cc285a2b4e315fb).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-894046589
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46645/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33646: [SPARK-36388][PYTHON] Fix DataFrame groupby-rolling to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-893213047
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46576/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33646: [SPARK-36388][PYTHON] Fix DataFrame groupby-rolling to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-893163413
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-894020209
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142133/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-894914281
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46713/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-894020209
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142133/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33646: [SPARK-36388][PYTHON] Fix DataFrame groupby-rolling to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-893174169
**[Test build #142066 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142066/testReport)** for PR 33646 at commit [`de59b6e`](https://github.com/apache/spark/commit/de59b6e8b293cfeb3b352ea05cc285a2b4e315fb).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-893966232
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46631/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-895664441
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46753/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin edited a comment on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
ueshin edited a comment on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-900582215
@itholic We forgot to modify tests in `python/pyspark/pandas/tests/test_ops_on_diff_frames_groupby_expanding.py` and `python/pyspark/pandas/tests/test_ops_on_diff_frames_groupby_rolling.py`.
Could you submit a follow-up PR to fix them?
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-894907498
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142201/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #33646:
URL: https://github.com/apache/spark/pull/33646#discussion_r685594303
##########
File path: python/pyspark/pandas/window.py
##########
@@ -706,10 +706,16 @@ def _apply_as_series_or_frame(self, func: Callable[[Column], Column]) -> FrameLi
if groupby._agg_columns_selected:
agg_columns = groupby._agg_columns
else:
+ # pandas doesn't keep the groupkey as a column from 1.3 for DataFrameGroupBy
+ if isinstance(groupby, DataFrameGroupBy):
+ for groupkey in groupby._groupkeys: # type: ignore
+ groupby._column_labels_to_exclude.add( # type: ignore
Review comment:
Oh, I got it. Let me address
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-894062194
**[Test build #142120 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142120/testReport)** for PR 33646 at commit [`f5267ad`](https://github.com/apache/spark/commit/f5267ad2dc822c8cd4f93d73ec3d47915c4dfe49).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-893948868
**[Test build #142121 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142121/testReport)** for PR 33646 at commit [`eb464f5`](https://github.com/apache/spark/commit/eb464f57f902b425609a7e7e4649adc07bc0ad9d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-893981752
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46634/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-894971443
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46719/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-895652387
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-894904735
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46713/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-906844217
Merged to branch-3.2 too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-895643177
**[Test build #142246 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142246/testReport)** for PR 33646 at commit [`c518cdd`](https://github.com/apache/spark/commit/c518cddb7464fbc1a6677469b8df345760a2288b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-893966232
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46631/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33646: [SPARK-36388][PYTHON] Fix DataFrame groupby-rolling to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-893207228
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46576/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33646:
URL: https://github.com/apache/spark/pull/33646#discussion_r683904589
##########
File path: python/pyspark/pandas/window.py
##########
@@ -706,11 +706,16 @@ def _apply_as_series_or_frame(self, func: Callable[[Column], Column]) -> FrameLi
if groupby._agg_columns_selected:
agg_columns = groupby._agg_columns
else:
- agg_columns = [
- psdf._psser_for(label)
+ agg_column_labels = [
+ label
for label in psdf._internal.column_labels
if label not in groupby._column_labels_to_exlcude
Review comment:
Can we add the column label to `column_labels_to_exlcude` (with some comments to explain why) at `DataFrameGroupBy` constructor?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
ueshin commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-900582215
@itholic We were missing to modify tests in `python/pyspark/pandas/tests/test_ops_on_diff_frames_groupby_expanding.py` and `python/pyspark/pandas/tests/test_ops_on_diff_frames_groupby_rolling.py`.
Could you submit a follow-up PR to fix them?
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-895672103
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46753/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin edited a comment on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
ueshin edited a comment on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-900582215
@itholic We forgot to modify tests in `python/pyspark/pandas/tests/test_ops_on_diff_frames_groupby_expanding.py` and `python/pyspark/pandas/tests/test_ops_on_diff_frames_groupby_rolling.py`.
Could you submit a follow-up PR to fix them?
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-894945197
**[Test build #142207 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142207/testReport)** for PR 33646 at commit [`a5c169a`](https://github.com/apache/spark/commit/a5c169a6dc3ca4ecdadae0beabac8565def7a4f8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-893946721
**[Test build #142120 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142120/testReport)** for PR 33646 at commit [`f5267ad`](https://github.com/apache/spark/commit/f5267ad2dc822c8cd4f93d73ec3d47915c4dfe49).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic edited a comment on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
itholic edited a comment on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-893947539
FYI: both SPARK-36388 and SPARK-36386 are resolved with same fix, so I consolidate them in single PR.
Also cc @ueshin @HyukjinKwon @xinrong-databricks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-894909570
@itholic can you check the test failures? https://github.com/itholic/spark/runs/3276195042?check_suite_focus=true
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-894914275
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46713/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
itholic commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-893947539
FYI: both SPARK-36388 and SPARK-36386 are resolved with same fix, so I consolidate them in this PR.
Also cc @ueshin @HyukjinKwon @xinrong-databricks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-894015793
**[Test build #142133 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142133/testReport)** for PR 33646 at commit [`4b1485e`](https://github.com/apache/spark/commit/4b1485e903f33e94feb349feaa705d52575bd729).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-895654929
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-894041025
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46645/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-894901360
**[Test build #142201 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142201/testReport)** for PR 33646 at commit [`11ea4a2`](https://github.com/apache/spark/commit/11ea4a241e5ab5c6f9ecdcbc4d6eb041712e6be7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33646: [SPARK-36388][PYTHON] Fix DataFrame groupby-rolling to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-893946721
**[Test build #142120 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142120/testReport)** for PR 33646 at commit [`f5267ad`](https://github.com/apache/spark/commit/f5267ad2dc822c8cd4f93d73ec3d47915c4dfe49).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-893960137
**[Test build #142121 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142121/testReport)** for PR 33646 at commit [`eb464f5`](https://github.com/apache/spark/commit/eb464f57f902b425609a7e7e4649adc07bc0ad9d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33646: [SPARK-36388][PYTHON] Fix DataFrame groupby-rolling to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-893184880
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-894046589
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46645/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-895633763
**[Test build #142246 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142246/testReport)** for PR 33646 at commit [`c518cdd`](https://github.com/apache/spark/commit/c518cddb7464fbc1a6677469b8df345760a2288b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #33646:
URL: https://github.com/apache/spark/pull/33646#discussion_r684020042
##########
File path: python/pyspark/pandas/window.py
##########
@@ -706,6 +706,10 @@ def _apply_as_series_or_frame(self, func: Callable[[Column], Column]) -> FrameLi
if groupby._agg_columns_selected:
agg_columns = groupby._agg_columns
else:
+ # pandas doesn't keep the groupkey as a column from 1.3 for DataFrameGroupBy
+ if isinstance(groupby, DataFrameGroupBy):
+ for groupkey in groupby._groupkeys: # type: ignore
+ groupby._column_labels_to_exlcude.add(groupkey._internal.column_labels[0])
Review comment:
Should we rename the from `exlcude` to `exclude` in this PR, too. ?? It's typo.
##########
File path: python/pyspark/pandas/window.py
##########
@@ -706,6 +706,10 @@ def _apply_as_series_or_frame(self, func: Callable[[Column], Column]) -> FrameLi
if groupby._agg_columns_selected:
agg_columns = groupby._agg_columns
else:
+ # pandas doesn't keep the groupkey as a column from 1.3 for DataFrameGroupBy
+ if isinstance(groupby, DataFrameGroupBy):
+ for groupkey in groupby._groupkeys: # type: ignore
+ groupby._column_labels_to_exlcude.add(groupkey._internal.column_labels[0])
Review comment:
btw, should we rename the from `exlcude` to `exclude` in this PR, too. ?? It's typo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
ueshin commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-900582215
@itholic We were missing to modify tests in `python/pyspark/pandas/tests/test_ops_on_diff_frames_groupby_expanding.py` and `python/pyspark/pandas/tests/test_ops_on_diff_frames_groupby_rolling.py`.
Could you submit a follow-up PR to fix them?
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33646:
URL: https://github.com/apache/spark/pull/33646#discussion_r684564692
##########
File path: python/pyspark/pandas/window.py
##########
@@ -706,6 +706,10 @@ def _apply_as_series_or_frame(self, func: Callable[[Column], Column]) -> FrameLi
if groupby._agg_columns_selected:
agg_columns = groupby._agg_columns
else:
+ # pandas doesn't keep the groupkey as a column from 1.3 for DataFrameGroupBy
+ if isinstance(groupby, DataFrameGroupBy):
+ for groupkey in groupby._groupkeys: # type: ignore
+ groupby._column_labels_to_exlcude.add(groupkey._internal.column_labels[0])
Review comment:
yeah please go ahead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #33646:
URL: https://github.com/apache/spark/pull/33646#discussion_r683905345
##########
File path: python/pyspark/pandas/window.py
##########
@@ -706,11 +706,16 @@ def _apply_as_series_or_frame(self, func: Callable[[Column], Column]) -> FrameLi
if groupby._agg_columns_selected:
agg_columns = groupby._agg_columns
else:
- agg_columns = [
- psdf._psser_for(label)
+ agg_column_labels = [
+ label
for label in psdf._internal.column_labels
if label not in groupby._column_labels_to_exlcude
Review comment:
Got it, and seems like we should also rename `exlcude` to `exclude` ??
It's typo 😮
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-893964197
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142121/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-894004368
**[Test build #142133 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142133/testReport)** for PR 33646 at commit [`4b1485e`](https://github.com/apache/spark/commit/4b1485e903f33e94feb349feaa705d52575bd729).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #33646:
URL: https://github.com/apache/spark/pull/33646#discussion_r685436498
##########
File path: python/pyspark/pandas/window.py
##########
@@ -706,10 +706,16 @@ def _apply_as_series_or_frame(self, func: Callable[[Column], Column]) -> FrameLi
if groupby._agg_columns_selected:
agg_columns = groupby._agg_columns
else:
+ # pandas doesn't keep the groupkey as a column from 1.3 for DataFrameGroupBy
+ if isinstance(groupby, DataFrameGroupBy):
+ for groupkey in groupby._groupkeys: # type: ignore
+ groupby._column_labels_to_exclude.add( # type: ignore
Review comment:
Shall we not update `groupby._column_labels_to_exclude` directly? The `groupby` might be reused.
```py
column_labels_to_exclude = groupby._column_labels_to_exclude.copy()
if isinstance(groupby, DataFrameGroupBy):
for ...
column_labels_to_exclude.add( ... )
agg_columns = [ ... if label not in column_labels_to_exclude]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33646:
URL: https://github.com/apache/spark/pull/33646#issuecomment-894936794
**[Test build #142207 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142207/testReport)** for PR 33646 at commit [`a5c169a`](https://github.com/apache/spark/commit/a5c169a6dc3ca4ecdadae0beabac8565def7a4f8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #33646: [SPARK-36388][SPARK-36386][PYTHON] Fix DataFrame groupby-rolling and groupby-expanding to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #33646:
URL: https://github.com/apache/spark/pull/33646
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org