You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/08/24 05:14:29 UTC
[GitHub] [spark] itholic opened a new pull request #33817: [SPARK-36537][PYTHON] Take care of other tests disabled for CategoricalDtype.
itholic opened a new pull request #33817:
URL: https://github.com/apache/spark/pull/33817
### What changes were proposed in this pull request?
This PR proposes to enable the tests, disabled since different behavior with pandas 1.3.
- `inplace` argument for CategoricalDtype functions is deprecated from pandas 1.3, and seems they have bug. So we manually created the expected result and test them.
- Fixed the `GroupBy.transform` since it doesn't work properly for `CategoricalDtype`.
### Why are the changes needed?
We should enable the tests as much as possible even if pandas has a bug.
And we should follow the behavior of latest pandas.
### Does this PR introduce _any_ user-facing change?
Yes, `GroupBy.transform` now follow the behavior of latest pandas.
### How was this patch tested?
Unittests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-906017746
**[Test build #142779 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142779/testReport)** for PR 33817 at commit [`bbb1501`](https://github.com/apache/spark/commit/bbb1501732197e73c8f2e9995e17c89a778d67a8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-906031409
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47277/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-905135401
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142745/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-906013435
**[Test build #142777 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142777/testReport)** for PR 33817 at commit [`c6418e9`](https://github.com/apache/spark/commit/c6418e998a4a6915a4bee08ba75cac4d07124713).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-906213551
Merged to master and branch-3.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-906017746
**[Test build #142779 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142779/testReport)** for PR 33817 at commit [`bbb1501`](https://github.com/apache/spark/commit/bbb1501732197e73c8f2e9995e17c89a778d67a8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33817: [SPARK-36537][PYTHON] Take care of other tests disabled for CategoricalDtype.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-904367766
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142716/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
itholic commented on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-906007065
Thanks for the correction, @ueshin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-906012879
**[Test build #142777 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142777/testReport)** for PR 33817 at commit [`c6418e9`](https://github.com/apache/spark/commit/c6418e998a4a6915a4bee08ba75cac4d07124713).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-906071248
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47279/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-906029460
**[Test build #142779 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142779/testReport)** for PR 33817 at commit [`bbb1501`](https://github.com/apache/spark/commit/bbb1501732197e73c8f2e9995e17c89a778d67a8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33817: [SPARK-36537][PYTHON] Take care of other tests disabled for CategoricalDtype.
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-904394725
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47216/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-906213551
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-905159080
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47245/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-906844275
Merged to branch-3.2 too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33817: [SPARK-36537][PYTHON] Take care of other tests disabled for CategoricalDtype.
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-904355890
**[Test build #142716 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142716/testReport)** for PR 33817 at commit [`10f6ff2`](https://github.com/apache/spark/commit/10f6ff2a5599868dbdb6bfe8fcb80dabe42f6fe3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33817: [SPARK-36537][PYTHON] Take care of other tests disabled for CategoricalDtype.
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-904363612
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47216/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33817: [SPARK-36537][PYTHON] Take care of other tests disabled for CategoricalDtype.
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-904341380
**[Test build #142716 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142716/testReport)** for PR 33817 at commit [`10f6ff2`](https://github.com/apache/spark/commit/10f6ff2a5599868dbdb6bfe8fcb80dabe42f6fe3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #33817:
URL: https://github.com/apache/spark/pull/33817
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #33817:
URL: https://github.com/apache/spark/pull/33817#discussion_r695233945
##########
File path: python/pyspark/pandas/tests/test_categorical.py
##########
@@ -214,17 +211,23 @@ def test_as_ordered_unordered(self):
pser.cat.as_ordered(inplace=True)
psser.cat.as_ordered(inplace=True)
if LooseVersion(pd.__version__) >= LooseVersion("1.3"):
- # TODO(SPARK-36367): Fix the behavior to follow pandas >= 1.3
- pass
- else:
- self.assert_eq(pser, psser)
+ # Bug in pandas 1.3. dtype is not updated properly with `inplace` argument.
+ pser = pser.astype(CategoricalDtype(categories=[1, 2, 3], ordered=True))
+
+ self.assert_eq(pser, psser)
+
Review comment:
nit: Shall we remove an extra line here?
##########
File path: python/pyspark/pandas/tests/test_categorical.py
##########
@@ -448,10 +451,7 @@ def identity(x) -> ps.Series[psdf.b.dtype]: # type: ignore
def astype(x) -> ps.Series[dtype]:
return x.astype(dtype)
- if LooseVersion(pd.__version__) >= LooseVersion("1.3"):
- # TODO(SPARK-36367): Fix the behavior to follow pandas >= 1.3
- pass
- elif LooseVersion(pd.__version__) >= LooseVersion("1.2"):
+ if LooseVersion(pd.__version__) >= LooseVersion("1.2"):
Review comment:
This behavior was change in pandas 1.3, right?
It seems to fail with pandas 1.2 now.
Also could you check with the older pandas?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-905135401
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142745/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-906049745
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-905159080
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47245/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-906050146
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47279/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-906049745
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-906029802
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47277/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33817: [SPARK-36537][PYTHON] Take care of other tests disabled for CategoricalDtype.
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-904341380
**[Test build #142716 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142716/testReport)** for PR 33817 at commit [`10f6ff2`](https://github.com/apache/spark/commit/10f6ff2a5599868dbdb6bfe8fcb80dabe42f6fe3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-906065707
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47279/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #33817:
URL: https://github.com/apache/spark/pull/33817#discussion_r696112596
##########
File path: python/pyspark/pandas/groupby.py
##########
@@ -2251,11 +2251,20 @@ def pandas_transform(pdf: pd.DataFrame) -> pd.DataFrame:
dtype = cast(SeriesType, return_type).dtype
spark_type = cast(SeriesType, return_type).spark_type
- data_fields = [
- InternalField(dtype=dtype, struct_field=StructField(name=c, dataType=spark_type))
- for c in psdf._internal.data_spark_column_names
- if c not in groupkey_names
- ]
+ data_fields = []
+ for c in psdf._internal.data_spark_column_names:
+ if c not in groupkey_names:
+ # If the unique values of CategoricalDtype are the same, use the original one.
+ # This behavior is changed from pandas 1.3.
+ original_dtype = psdf[c].dtype
+ if isinstance(original_dtype, CategoricalDtype) and original_dtype == dtype:
+ dtype = original_dtype
+ data_fields.append(
+ InternalField(
+ dtype=dtype, struct_field=StructField(name=c, dataType=spark_type)
+ )
+ )
+
Review comment:
On second thoughts, I think this update is wrong.
##########
File path: python/pyspark/pandas/tests/test_categorical.py
##########
@@ -443,15 +445,16 @@ def identity(x) -> ps.Series[psdf.b.dtype]: # type: ignore
pdf.groupby("a").transform(identity).sort_values("b").reset_index(drop=True),
)
- dtype = CategoricalDtype(categories=["a", "b", "c", "d"])
+ # The behavior for CategoricalDtype is changed from pandas 1.3
+ if LooseVersion(pd.__version__) >= LooseVersion("1.3"):
+ dtype = CategoricalDtype(categories=["a", "b", "c", "d"])
+ else:
+ dtype = pdf.b.dtype
def astype(x) -> ps.Series[dtype]:
return x.astype(dtype)
Review comment:
Instead, the return type of this function `astype` should be the original dtype instead of the new one.
so, I think this should be:
```py
dtype = CategoricalDtype(categories=["a", "b", "c", "d"])
# The behavior for CategoricalDtype is changed from pandas 1.3
if LooseVersion(pd.__version__) >= LooseVersion("1.3"):
ret_dtype = pdf.b.dtype
else:
ret_dtype = dtype
def astype(x) -> ps.Series[ret_dtype]:
return x.astype(dtype)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-905117841
**[Test build #142745 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142745/testReport)** for PR 33817 at commit [`c272351`](https://github.com/apache/spark/commit/c2723515ac455aa51f46ffaffd861757d75b9461).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33817: [SPARK-36537][PYTHON] Take care of other tests disabled for CategoricalDtype.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-904367766
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142716/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-906071248
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47279/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-905105478
**[Test build #142745 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142745/testReport)** for PR 33817 at commit [`c272351`](https://github.com/apache/spark/commit/c2723515ac455aa51f46ffaffd861757d75b9461).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33817: [SPARK-36537][PYTHON] Take care of other tests disabled for CategoricalDtype.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-904404362
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47216/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #33817:
URL: https://github.com/apache/spark/pull/33817#discussion_r695330584
##########
File path: python/pyspark/pandas/tests/test_categorical.py
##########
@@ -448,10 +451,7 @@ def identity(x) -> ps.Series[psdf.b.dtype]: # type: ignore
def astype(x) -> ps.Series[dtype]:
return x.astype(dtype)
- if LooseVersion(pd.__version__) >= LooseVersion("1.3"):
- # TODO(SPARK-36367): Fix the behavior to follow pandas >= 1.3
- pass
- elif LooseVersion(pd.__version__) >= LooseVersion("1.2"):
+ if LooseVersion(pd.__version__) >= LooseVersion("1.2"):
Review comment:
Thanks! Just fixed tests for pandas 1.0, 1.1, 1.2 series.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-906012879
**[Test build #142777 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142777/testReport)** for PR 33817 at commit [`c6418e9`](https://github.com/apache/spark/commit/c6418e998a4a6915a4bee08ba75cac4d07124713).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-906013451
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142777/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-905105478
**[Test build #142745 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142745/testReport)** for PR 33817 at commit [`c272351`](https://github.com/apache/spark/commit/c2723515ac455aa51f46ffaffd861757d75b9461).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-906013451
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142777/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33817: [SPARK-36537][PYTHON] Take care of other tests disabled for CategoricalDtype.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-904404362
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47216/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-905132011
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47245/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33817: [SPARK-36537][PYTHON] Revisit disabled tests for CategoricalDtype
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33817:
URL: https://github.com/apache/spark/pull/33817#issuecomment-905148097
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47245/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org