You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/07/28 23:27:53 UTC
[GitHub] [spark] ueshin opened a new pull request #33562: [SPARK-36333][PYTHON] Reuse isnull where the nullability check is needed
ueshin opened a new pull request #33562:
URL: https://github.com/apache/spark/pull/33562
### What changes were proposed in this pull request?
Reuse `IndexOpsMixin.isnull()` where the nullability check is needed.
### Why are the changes needed?
There are some places where we can reuse `IndexOpsMixin.isnull()` instead of directly using Spark `Column`.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33562: [SPARK-36333][PYTHON] Reuse isnull where the null check is needed
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33562:
URL: https://github.com/apache/spark/pull/33562#issuecomment-889509732
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46368/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33562: [SPARK-36333][PYTHON] Reuse isnull where the null check is needed
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33562:
URL: https://github.com/apache/spark/pull/33562#issuecomment-888686939
**[Test build #141793 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141793/testReport)** for PR 33562 at commit [`21708a0`](https://github.com/apache/spark/commit/21708a00f52030215de09a21e38f2febfcc23bf6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33562: [SPARK-36333][PYTHON] Reuse isnull where the null check is needed
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33562:
URL: https://github.com/apache/spark/pull/33562#issuecomment-889509732
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46368/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33562: [SPARK-36333][PYTHON] Reuse isnull where the null check is needed
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33562:
URL: https://github.com/apache/spark/pull/33562#issuecomment-888728933
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46306/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33562: [SPARK-36333][PYTHON] Reuse isnull where the null check is needed
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33562:
URL: https://github.com/apache/spark/pull/33562#issuecomment-888698801
**[Test build #141793 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141793/testReport)** for PR 33562 at commit [`21708a0`](https://github.com/apache/spark/commit/21708a00f52030215de09a21e38f2febfcc23bf6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33562: [SPARK-36333][PYTHON] Reuse isnull where the null check is needed
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33562:
URL: https://github.com/apache/spark/pull/33562#issuecomment-889474670
**[Test build #141857 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141857/testReport)** for PR 33562 at commit [`3ea75f4`](https://github.com/apache/spark/commit/3ea75f4960825310076cdd6897937319ac8e26a1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33562: [SPARK-36333][PYTHON] Reuse isnull where the null check is needed
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33562:
URL: https://github.com/apache/spark/pull/33562#issuecomment-888703156
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46306/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin closed pull request #33562: [SPARK-36333][PYTHON] Reuse isnull where the null check is needed
Posted by GitBox <gi...@apache.org>.
ueshin closed pull request #33562:
URL: https://github.com/apache/spark/pull/33562
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33562: [SPARK-36333][PYTHON] Reuse isnull where the null check is needed
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33562:
URL: https://github.com/apache/spark/pull/33562#issuecomment-888707198
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141793/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33562: [SPARK-36333][PYTHON] Reuse isnull where the null check is needed
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33562:
URL: https://github.com/apache/spark/pull/33562#issuecomment-888686939
**[Test build #141793 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141793/testReport)** for PR 33562 at commit [`21708a0`](https://github.com/apache/spark/commit/21708a00f52030215de09a21e38f2febfcc23bf6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33562: [SPARK-36333][PYTHON] Reuse isnull where the null check is needed
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33562:
URL: https://github.com/apache/spark/pull/33562#issuecomment-889487698
**[Test build #141857 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141857/testReport)** for PR 33562 at commit [`3ea75f4`](https://github.com/apache/spark/commit/3ea75f4960825310076cdd6897937319ac8e26a1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33562: [SPARK-36333][PYTHON] Reuse isnull where the null check is needed
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33562:
URL: https://github.com/apache/spark/pull/33562#issuecomment-889474670
**[Test build #141857 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141857/testReport)** for PR 33562 at commit [`3ea75f4`](https://github.com/apache/spark/commit/3ea75f4960825310076cdd6897937319ac8e26a1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #33562: [SPARK-36333][PYTHON] Reuse isnull where the null check is needed
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #33562:
URL: https://github.com/apache/spark/pull/33562#discussion_r678749415
##########
File path: python/pyspark/pandas/generic.py
##########
@@ -3180,6 +3180,7 @@ def __bool__(self) -> NoReturn:
def _count_expr(spark_column: Column, spark_type: DataType) -> Column:
# Special handle floating point types because Spark's count treats nan as a valid value,
# whereas pandas count doesn't include nan.
+ # TODO: Make this work with DataTypeOps.
Review comment:
Maybe we want to create a related ticket for TODO and comment with ticket number ?
e.g. `TODO(SPARK-xxxxx) Make this work with DataTypeOps.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33562: [SPARK-36333][PYTHON] Reuse isnull where the null check is needed
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33562:
URL: https://github.com/apache/spark/pull/33562#issuecomment-888715825
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46306/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33562: [SPARK-36333][PYTHON] Reuse isnull where the null check is needed
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33562:
URL: https://github.com/apache/spark/pull/33562#issuecomment-889502950
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141857/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on pull request #33562: [SPARK-36333][PYTHON] Reuse isnull where the null check is needed
Posted by GitBox <gi...@apache.org>.
ueshin commented on pull request #33562:
URL: https://github.com/apache/spark/pull/33562#issuecomment-889503820
Thanks! merging to master/3.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33562: [SPARK-36333][PYTHON] Reuse isnull where the null check is needed
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33562:
URL: https://github.com/apache/spark/pull/33562#issuecomment-889509718
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46368/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33562: [SPARK-36333][PYTHON] Reuse isnull where the null check is needed
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33562:
URL: https://github.com/apache/spark/pull/33562#issuecomment-889496518
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46368/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33562: [SPARK-36333][PYTHON] Reuse isnull where the null check is needed
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33562:
URL: https://github.com/apache/spark/pull/33562#issuecomment-888707198
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141793/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #33562: [SPARK-36333][PYTHON] Reuse isnull where the null check is needed
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #33562:
URL: https://github.com/apache/spark/pull/33562#discussion_r678749415
##########
File path: python/pyspark/pandas/generic.py
##########
@@ -3180,6 +3180,7 @@ def __bool__(self) -> NoReturn:
def _count_expr(spark_column: Column, spark_type: DataType) -> Column:
# Special handle floating point types because Spark's count treats nan as a valid value,
# whereas pandas count doesn't include nan.
+ # TODO: Make this work with DataTypeOps.
Review comment:
Maybe we want to create a related ticket for TODO and comment with ticket number ?
e.g. `TODO(SPARK-xxxxx): Make this work with DataTypeOps.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on pull request #33562: [SPARK-36333][PYTHON] Reuse isnull where the nullability check is needed
Posted by GitBox <gi...@apache.org>.
ueshin commented on pull request #33562:
URL: https://github.com/apache/spark/pull/33562#issuecomment-888685711
cc @xinrong-databricks @HyukjinKwon @itholic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33562: [SPARK-36333][PYTHON] Reuse isnull where the null check is needed
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33562:
URL: https://github.com/apache/spark/pull/33562#issuecomment-889502950
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141857/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33562: [SPARK-36333][PYTHON] Reuse isnull where the null check is needed
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33562:
URL: https://github.com/apache/spark/pull/33562#issuecomment-888728933
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46306/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org