You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/05/20 18:30:49 UTC
[GitHub] [spark] xinrong-databricks opened a new pull request #32611: [WIP][SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
xinrong-databricks opened a new pull request #32611:
URL: https://github.com/apache/spark/pull/32611
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error message, please read the guideline first:
https://spark.apache.org/error-message-guidelines.html
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847346961
**[Test build #138885 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138885/testReport)** for PR 32611 at commit [`5e2f911`](https://github.com/apache/spark/commit/5e2f91152dc0ad45c7155d94e4b87e82523ed04b).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850725471
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43594/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r637299344
##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +41,183 @@ class BooleanOps(DataTypeOps):
@property
def pretty_name(self) -> str:
return 'booleans'
+
+ def add(self, left, right):
+ if isinstance(right, numbers.Number):
+ left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
+ return column_op(Column.__add__)(left, right)
+ elif isinstance(right, IndexOpsMixin) and is_numeric_index_ops(right):
+ left = transform_boolean_operand_to_numeric(left, right.spark.data_type)
+ return column_op(Column.__add__)(left, right)
+ else:
+ raise TypeError(
+ "Addition can not be applied to %s and the given type." % self.pretty_name)
+
+ def sub(self, left, right):
+ if isinstance(right, numbers.Number):
+ left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
+ return column_op(Column.__sub__)(left, right)
+ elif isinstance(right, IndexOpsMixin) and is_numeric_index_ops(right):
+ left = transform_boolean_operand_to_numeric(left, right.spark.data_type)
+ return column_op(Column.__sub__)(left, right)
+ else:
+ raise TypeError(
+ "Subtraction can not be applied to %s and the given type." % self.pretty_name)
+
+ def mul(self, left, right):
+ if isinstance(right, numbers.Number):
+ left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
+ return column_op(Column.__mul__)(left, right)
+ elif isinstance(right, IndexOpsMixin) and is_numeric_index_ops(right):
+ left = transform_boolean_operand_to_numeric(left, right.spark.data_type)
+ return column_op(Column.__mul__)(left, right)
+ else:
+ raise TypeError(
+ "Multiplication can not be applied to %s and the given type." % self.pretty_name)
+
+ def truediv(self, left, right):
+ def truediv(left, right):
+ return F.when(F.lit(right != 0) | F.lit(right).isNull(), left.__div__(right)).otherwise(
+ F.when(F.lit(left == np.inf) | F.lit(left == -np.inf), left).otherwise(
+ F.lit(np.inf).__div__(left)
+ )
+ )
+
+ if isinstance(right, numbers.Number):
+ left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
+ return numpy_column_op(truediv)(left, right)
+ elif isinstance(right, IndexOpsMixin) and is_numeric_index_ops(right):
+ left = transform_boolean_operand_to_numeric(left, right.spark.data_type)
+ return numpy_column_op(truediv)(left, right)
+ else:
+ raise TypeError(
+ "True division can not be applied to %s and the given type." % self.pretty_name)
+
+ def floordiv(self, left, right):
+ def floordiv(left, right):
+ return F.when(F.lit(right is np.nan), np.nan).otherwise(
+ F.when(
+ F.lit(right != 0) | F.lit(right).isNull(), F.floor(left.__div__(right))
+ ).otherwise(
+ F.when(F.lit(left == np.inf) | F.lit(left == -np.inf), left).otherwise(
+ F.lit(np.inf).__div__(left)
+ )
+ )
+ )
+
+ if isinstance(right, numbers.Number):
+ left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
+ return numpy_column_op(floordiv)(left, right)
+ elif isinstance(right, IndexOpsMixin) and is_numeric_index_ops(right):
+ left = transform_boolean_operand_to_numeric(left, right.spark.data_type)
+ return numpy_column_op(floordiv)(left, right)
+ else:
+ raise TypeError(
+ "Floor division can not be applied to %s and the given type." % self.pretty_name)
+
+ def mod(self, left, right):
+ def mod(left, right):
+ return ((left % right) + right) % right
+
+ if isinstance(right, numbers.Number):
+ left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
+ return column_op(mod)(left, right)
+ elif isinstance(right, IndexOpsMixin) and is_numeric_index_ops(right):
+ left = transform_boolean_operand_to_numeric(left, right.spark.data_type)
+ return column_op(mod)(left, right)
+ else:
+ raise TypeError(
+ "Modulo can not be applied to %s and the given type." % self.pretty_name)
+
+ def pow(self, left, right):
+ def pow_func(left, right):
+ return F.when(left == 1, left).otherwise(Column.__pow__(left, right))
+
+ if isinstance(right, numbers.Number):
+ left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
+ return column_op(pow_func)(left, right)
+ elif isinstance(right, IndexOpsMixin) and is_numeric_index_ops(right):
+ left = transform_boolean_operand_to_numeric(left, right.spark.data_type)
+ return column_op(pow_func)(left, right)
+
+ else:
+ raise TypeError(
+ "Exponentiation can not be applied to %s and the given type." % self.pretty_name)
+
+ def radd(self, left, right=None):
+ if isinstance(right, numbers.Number):
+ left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
+ return column_op(Column.__radd__)(left, right)
+ else:
+ raise TypeError(
+ "Addition can not be applied to %s and the given type." % self.pretty_name)
+
+ def rsub(self, left, right=None):
Review comment:
`=None`s are removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849985493
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43551/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845577570
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43289/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849197264
**[Test build #138991 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138991/testReport)** for PR 32611 at commit [`5905012`](https://github.com/apache/spark/commit/5905012338a16ba500094b9cae33bb783e86faff).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [WIP][SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845505250
**[Test build #138766 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138766/testReport)** for PR 32611 at commit [`3f0fc27`](https://github.com/apache/spark/commit/3f0fc27ffe093ff6ef12c3f5214f6fa98c635860).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847346980
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138885/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847263424
**[Test build #138882 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138882/testReport)** for PR 32611 at commit [`43a2972`](https://github.com/apache/spark/commit/43a29721e982df716a874286347e8b93492e1fd7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r638330196
##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +38,136 @@ class BooleanOps(DataTypeOps):
@property
def pretty_name(self) -> str:
return 'booleans'
+
+ def add(self, left, right) -> Union["Series", "Index"]:
+ if isinstance(right, numbers.Number):
Review comment:
We should disallow `bool` literal?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847466995
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43417/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850542913
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139066/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847429108
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43410/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847293372
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43404/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r637299114
##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +41,183 @@ class BooleanOps(DataTypeOps):
@property
def pretty_name(self) -> str:
return 'booleans'
+
+ def add(self, left, right):
Review comment:
For sure, type annotations are added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [WIP][SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845431129
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138763/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849180579
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43509/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r637338960
##########
File path: python/pyspark/pandas/tests/data_type_ops/test_boolean_ops.py
##########
@@ -33,107 +35,148 @@ def pser(self):
def psser(self):
return ps.from_pandas(self.pser)
+ @property
+ def float_pser(self):
+ return pd.Series([1, 2, 3], dtype=float)
+
+ @property
+ def float_psser(self):
+ return ps.from_pandas(self.float_pser)
+
def test_add(self):
- self.assertRaises(TypeError, lambda: self.psser + 1)
- self.assertRaises(TypeError, lambda: self.psser + 0.1)
+ self.assert_eq(self.pser + 1, self.psser + 1)
+ self.assert_eq(self.pser + 0.1, self.psser + 0.1)
with option_context("compute.ops_on_diff_frames", True):
- for psser in self.pssers:
+ for pser, psser in self.numeric_pser_psser_pairs:
+ self.assert_eq(self.pser + pser, (self.psser + psser).sort_index())
+
+ for psser in self.non_numeric_pssers.values():
self.assertRaises(TypeError, lambda: self.psser + psser)
def test_sub(self):
- self.assertRaises(TypeError, lambda: self.psser - 1)
- self.assertRaises(TypeError, lambda: self.psser - 0.1)
+ self.assert_eq(self.pser - 1, self.psser - 1)
+ self.assert_eq(self.pser - 0.1, self.psser - 0.1)
with option_context("compute.ops_on_diff_frames", True):
- for psser in self.pssers:
+ for pser, psser in self.numeric_pser_psser_pairs:
+ self.assert_eq(self.pser - pser, (self.psser - psser).sort_index())
+
+ for psser in self.non_numeric_pssers.values():
self.assertRaises(TypeError, lambda: self.psser - psser)
def test_mul(self):
- self.assertRaises(TypeError, lambda: self.psser * 1)
- self.assertRaises(TypeError, lambda: self.psser * 0.1)
+ self.assert_eq(self.pser * 1, self.psser * 1)
+ self.assert_eq(self.pser * 0.1, self.psser * 0.1)
with option_context("compute.ops_on_diff_frames", True):
- for psser in self.pssers:
+ for pser, psser in self.numeric_pser_psser_pairs:
+ self.assert_eq(self.pser * pser, (self.psser * psser).sort_index())
+
+ for psser in self.non_numeric_pssers.values():
self.assertRaises(TypeError, lambda: self.psser * psser)
def test_truediv(self):
- self.assertRaises(TypeError, lambda: self.psser / 1)
- self.assertRaises(TypeError, lambda: self.psser / 0.1)
+ self.assert_eq(self.pser / 1, self.psser / 1)
+ self.assert_eq(self.pser / 0.1, self.psser / 0.1)
with option_context("compute.ops_on_diff_frames", True):
- for psser in self.pssers:
+ self.assert_eq(
+ self.pser / self.float_pser, (self.psser / self.float_psser).sort_index())
+
+ for psser in self.non_numeric_pssers.values():
self.assertRaises(TypeError, lambda: self.psser / psser)
def test_floordiv(self):
- self.assertRaises(TypeError, lambda: self.psser // 1)
- self.assertRaises(TypeError, lambda: self.psser // 0.1)
+ # float is always returned in pandas-on-Spark
+ self.assert_eq((self.pser // 1).astype("float"), self.psser // 1)
+ # in pandas, 1 // 0.1 = 9.0; in pandas-on-Spark, 1 // 0.1 = 10.0
+ # self.assert_eq(self.pser // 0.1, self.psser // 0.1)
with option_context("compute.ops_on_diff_frames", True):
- for psser in self.pssers:
+ self.assert_eq(
+ self.pser // self.float_pser, (self.psser // self.float_psser).sort_index()
+ )
+
+ for psser in self.non_numeric_pssers.values():
self.assertRaises(TypeError, lambda: self.psser // psser)
def test_mod(self):
- self.assertRaises(TypeError, lambda: self.psser % 1)
- self.assertRaises(TypeError, lambda: self.psser % 0.1)
+ self.assert_eq(self.pser % 1, self.psser % 1)
+ self.assert_eq(self.pser % 0.1, self.psser % 0.1)
with option_context("compute.ops_on_diff_frames", True):
- for psser in self.pssers:
+ for pser, psser in self.numeric_pser_psser_pairs:
+ self.assert_eq(self.pser % pser, (self.psser % psser).sort_index())
+
+ for psser in self.non_numeric_pssers.values():
self.assertRaises(TypeError, lambda: self.psser % psser)
def test_pow(self):
- self.assertRaises(TypeError, lambda: self.psser ** 1)
- self.assertRaises(TypeError, lambda: self.psser ** 0.1)
+ # float is always returned in pandas-on-Spark
+ self.assert_eq((self.pser ** 1).astype("float"), self.psser ** 1)
+ self.assert_eq(self.pser ** 0.1, self.psser ** 0.1)
with option_context("compute.ops_on_diff_frames", True):
- for psser in self.pssers:
+ self.assert_eq(
+ self.pser ** self.float_pser, (self.psser ** self.float_psser).sort_index()
+ )
+
+ for psser in self.non_numeric_pssers.values():
self.assertRaises(TypeError, lambda: self.psser ** psser)
def test_radd(self):
- self.assertRaises(TypeError, lambda: 1 + self.psser)
- self.assertRaises(TypeError, lambda: 0.1 + self.psser)
+ self.assert_eq(1 + self.pser, 1 + self.psser)
+ self.assert_eq(0.1 + self.pser, 0.1 + self.psser)
self.assertRaises(TypeError, lambda: "x" + self.psser)
self.assertRaises(TypeError, lambda: datetime.date(1994, 1, 1) + self.psser)
self.assertRaises(TypeError, lambda: datetime.datetime(1994, 1, 1) + self.psser)
def test_rsub(self):
- self.assertRaises(TypeError, lambda: 1 - self.psser)
- self.assertRaises(TypeError, lambda: 0.1 - self.psser)
+ self.assert_eq(1 - self.pser, 1 - self.psser)
+ self.assert_eq(0.1 - self.pser, 0.1 - self.psser)
self.assertRaises(TypeError, lambda: "x" - self.psser)
self.assertRaises(TypeError, lambda: datetime.date(1994, 1, 1) - self.psser)
self.assertRaises(TypeError, lambda: datetime.datetime(1994, 1, 1) - self.psser)
def test_rmul(self):
- self.assertRaises(TypeError, lambda: 1 * self.psser)
- self.assertRaises(TypeError, lambda: 0.1 * self.psser)
+ self.assert_eq(1 * self.pser, 1 * self.psser)
+ self.assert_eq(0.1 * self.pser, 0.1 * self.psser)
self.assertRaises(TypeError, lambda: "x" * self.psser)
self.assertRaises(TypeError, lambda: datetime.date(1994, 1, 1) * self.psser)
self.assertRaises(TypeError, lambda: datetime.datetime(1994, 1, 1) * self.psser)
def test_rtruediv(self):
- self.assertRaises(TypeError, lambda: 1 / self.psser)
- self.assertRaises(TypeError, lambda: 0.1 / self.psser)
+ self.assert_eq(1 / self.pser, 1 / self.psser)
+ self.assert_eq(0.1 / self.pser, 0.1 / self.psser)
self.assertRaises(TypeError, lambda: "x" / self.psser)
self.assertRaises(TypeError, lambda: datetime.date(1994, 1, 1) / self.psser)
self.assertRaises(TypeError, lambda: datetime.datetime(1994, 1, 1) / self.psser)
def test_rfloordiv(self):
- self.assertRaises(TypeError, lambda: 1 // self.psser)
- self.assertRaises(TypeError, lambda: 0.1 // self.psser)
+ if LooseVersion(pd.__version__) >= LooseVersion("0.25.3"):
+ self.assert_eq(1 // self.pser, 1 // self.psser)
+ self.assert_eq(0.1 // self.pser, 0.1 // self.psser)
Review comment:
What happens in the `else` case?
##########
File path: python/pyspark/pandas/tests/data_type_ops/test_num_ops.py
##########
@@ -101,7 +133,13 @@ def test_floordiv(self):
self.assertRaises(TypeError, lambda: psser // self.non_numeric_pssers["date"])
self.assertRaises(
TypeError, lambda: psser // self.non_numeric_pssers["categorical"])
- self.assertRaises(TypeError, lambda: psser // self.non_numeric_pssers["bool"])
+ if LooseVersion(pd.__version__) >= LooseVersion("0.25.3"):
+ self.assert_eq(
+ (self.float_psser // self.non_numeric_pssers["bool"]).sort_index(),
+ self.float_pser // self.non_numeric_psers["bool"],
+ )
Review comment:
ditto.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r638268484
##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -50,7 +50,7 @@ def pretty_name(self) -> str:
def add(self, left, right) -> Union["Series", "Index"]:
if isinstance(right, numbers.Number):
left = left.spark.transform(lambda scol: scol.cast(as_spark_type(type(right))))
- return column_op(Column.__add__)(left, right)
+ return left + right
elif isinstance(right, IndexOpsMixin) and is_numeric_index_ops(right):
left = transform_boolean_operand_to_numeric(left, right.spark.data_type)
return column_op(Column.__add__)(left, right)
Review comment:
You are right, I was thinking it caused an infinite recursion. `left + right`s are applied.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849193160
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43509/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847209831
**[Test build #138880 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138880/testReport)** for PR 32611 at commit [`b4e2bc7`](https://github.com/apache/spark/commit/b4e2bc7689cdac95e4358e48754f8cdb4cf313fb).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r638106867
##########
File path: python/pyspark/pandas/data_type_ops/num_ops.py
##########
@@ -16,31 +16,46 @@
#
import numbers
-from typing import TYPE_CHECKING, Union
+from typing import Any, TYPE_CHECKING, Union
import numpy as np
from pandas.api.types import CategoricalDtype
from pyspark.sql import Column, functions as F
from pyspark.sql.types import (
+ BooleanType,
NumericType,
StringType,
TimestampType,
)
from pyspark.pandas.base import column_op, IndexOpsMixin, numpy_column_op
-from pyspark.pandas.data_type_ops.base import DataTypeOps
+from pyspark.pandas.data_type_ops.base import DataTypeOps, transform_boolean_operand_to_numeric
from pyspark.pandas.spark import functions as SF
+from pyspark.sql.column import Column
+
if TYPE_CHECKING:
from pyspark.pandas.indexes import Index # noqa: F401 (SPARK-34943)
from pyspark.pandas.series import Series # noqa: F401 (SPARK-34943)
+def is_valid_operand_for_numeric_arithmetic(operand: Any) -> bool:
+ """Check whether the operand is valid for arithmetic operations against numerics."""
+ if isinstance(operand, numbers.Number):
+ return True
+ elif isinstance(operand, IndexOpsMixin):
+ if isinstance(operand.dtype, CategoricalDtype):
+ return False
+ else:
+ return isinstance(operand.spark.data_type, NumericType) or isinstance(
+ operand.spark.data_type, BooleanType)
+ else:
+ return isinstance(operand, Column)
Review comment:
Yes, we do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846081840
**[Test build #138809 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138809/testReport)** for PR 32611 at commit [`a123db8`](https://github.com/apache/spark/commit/a123db80c882b806c6c60fb56e83b8b5253e0b63).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r640174272
##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +39,144 @@ class BooleanOps(DataTypeOps):
@property
def pretty_name(self) -> str:
return 'booleans'
+
+ def add(self, left, right) -> Union["Series", "Index"]:
+ if not is_valid_operand_for_numeric_arithmetic(right, allow_bool=False):
+ raise TypeError(
+ "Addition can not be applied to %s and the given type." % self.pretty_name)
+
+ if isinstance(right, numbers.Number) and not isinstance(right, bool):
+ left = left.spark.transform(lambda scol: scol.cast(as_spark_type(type(right))))
+ return left + right
+ else:
+ assert isinstance(right, IndexOpsMixin)
Review comment:
Now `spark.Column` is disallowed in `is_valid_operand_for_numeric_arithmetic`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-848083906
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43457/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847427963
**[Test build #138891 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138891/testReport)** for PR 32611 at commit [`7ea2876`](https://github.com/apache/spark/commit/7ea2876d0c5083ffa2eddbc634edab6e4a74d71a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846321739
**[Test build #138818 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138818/testReport)** for PR 32611 at commit [`925e34b`](https://github.com/apache/spark/commit/925e34b3bffb6496d62425fa08d36177589cb17d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846080849
**[Test build #138809 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138809/testReport)** for PR 32611 at commit [`a123db8`](https://github.com/apache/spark/commit/a123db80c882b806c6c60fb56e83b8b5253e0b63).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [WIP][SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845470209
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43285/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845619671
@HyukjinKwon Certainly, examples are added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r638207153
##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -50,7 +50,7 @@ def pretty_name(self) -> str:
def add(self, left, right) -> Union["Series", "Index"]:
if isinstance(right, numbers.Number):
left = left.spark.transform(lambda scol: scol.cast(as_spark_type(type(right))))
- return column_op(Column.__add__)(left, right)
+ return left + right
elif isinstance(right, IndexOpsMixin) and is_numeric_index_ops(right):
left = transform_boolean_operand_to_numeric(left, right.spark.data_type)
return column_op(Column.__add__)(left, right)
Review comment:
We can use `left + right` here as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r637323275
##########
File path: python/pyspark/pandas/tests/data_type_ops/test_num_ops.py
##########
@@ -42,7 +54,12 @@ def test_add(self):
self.assertRaises(TypeError, lambda: psser + self.non_numeric_pssers["datetime"])
self.assertRaises(TypeError, lambda: psser + self.non_numeric_pssers["date"])
self.assertRaises(TypeError, lambda: psser + self.non_numeric_pssers["categorical"])
- self.assertRaises(TypeError, lambda: psser + self.non_numeric_pssers["bool"])
+ self.assert_eq(
+ (psser + self.non_numeric_pssers["bool"]).sort_index(),
+ pser + self.non_numeric_psers["bool"],
+ )
+
+ self.assertRaises(TypeError, lambda: self.float_psser + True)
Review comment:
This is supposed to be out of for-loop?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r637311351
##########
File path: python/pyspark/pandas/tests/data_type_ops/test_num_ops.py
##########
@@ -30,6 +34,14 @@ class NumOpsTest(PandasOnSparkTestCase, TestCasesUtils):
returns float32.
The underlying reason is the respective Spark operations return DoubleType always.
"""
+ @property
+ def float_pser(self):
+ return pd.Series([1, 2, 3], dtype=float)
+
+ @property
+ def float_psser(self):
+ return ps.from_pandas(self.float_pser)
+
def test_add(self):
for pser, psser in self.numeric_pser_psser_pairs:
self.assert_eq(pser + pser, psser + psser)
Review comment:
Isn't this PR supposed to support it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r637228133
##########
File path: python/pyspark/pandas/tests/data_type_ops/test_date_ops.py
##########
@@ -60,7 +61,9 @@ def test_sub(self):
with option_context("compute.ops_on_diff_frames", True):
for pser, psser in self.pser_psser_pairs:
if isinstance(psser.spark.data_type, DateType):
- self.assert_eq((self.pser - pser).dt.days, (self.psser - psser).sort_index())
+ if LooseVersion(pd.__version__) >= LooseVersion("0.24.2"):
+ self.assert_eq(
+ (self.pser - pser).dt.days, (self.psser - psser).sort_index())
Review comment:
What happens in `else` case?
##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +41,183 @@ class BooleanOps(DataTypeOps):
@property
def pretty_name(self) -> str:
return 'booleans'
+
+ def add(self, left, right):
+ if isinstance(right, numbers.Number):
+ left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
+ return column_op(Column.__add__)(left, right)
+ elif isinstance(right, IndexOpsMixin) and is_numeric_index_ops(right):
+ left = transform_boolean_operand_to_numeric(left, right.spark.data_type)
+ return column_op(Column.__add__)(left, right)
+ else:
+ raise TypeError(
+ "Addition can not be applied to %s and the given type." % self.pretty_name)
+
+ def sub(self, left, right):
+ if isinstance(right, numbers.Number):
+ left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
+ return column_op(Column.__sub__)(left, right)
+ elif isinstance(right, IndexOpsMixin) and is_numeric_index_ops(right):
+ left = transform_boolean_operand_to_numeric(left, right.spark.data_type)
+ return column_op(Column.__sub__)(left, right)
+ else:
+ raise TypeError(
+ "Subtraction can not be applied to %s and the given type." % self.pretty_name)
+
+ def mul(self, left, right):
+ if isinstance(right, numbers.Number):
+ left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
+ return column_op(Column.__mul__)(left, right)
+ elif isinstance(right, IndexOpsMixin) and is_numeric_index_ops(right):
+ left = transform_boolean_operand_to_numeric(left, right.spark.data_type)
+ return column_op(Column.__mul__)(left, right)
+ else:
+ raise TypeError(
+ "Multiplication can not be applied to %s and the given type." % self.pretty_name)
+
+ def truediv(self, left, right):
+ def truediv(left, right):
+ return F.when(F.lit(right != 0) | F.lit(right).isNull(), left.__div__(right)).otherwise(
+ F.when(F.lit(left == np.inf) | F.lit(left == -np.inf), left).otherwise(
+ F.lit(np.inf).__div__(left)
+ )
+ )
+
+ if isinstance(right, numbers.Number):
+ left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
+ return numpy_column_op(truediv)(left, right)
+ elif isinstance(right, IndexOpsMixin) and is_numeric_index_ops(right):
+ left = transform_boolean_operand_to_numeric(left, right.spark.data_type)
+ return numpy_column_op(truediv)(left, right)
+ else:
+ raise TypeError(
+ "True division can not be applied to %s and the given type." % self.pretty_name)
+
+ def floordiv(self, left, right):
+ def floordiv(left, right):
+ return F.when(F.lit(right is np.nan), np.nan).otherwise(
+ F.when(
+ F.lit(right != 0) | F.lit(right).isNull(), F.floor(left.__div__(right))
+ ).otherwise(
+ F.when(F.lit(left == np.inf) | F.lit(left == -np.inf), left).otherwise(
+ F.lit(np.inf).__div__(left)
+ )
+ )
+ )
+
+ if isinstance(right, numbers.Number):
+ left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
+ return numpy_column_op(floordiv)(left, right)
+ elif isinstance(right, IndexOpsMixin) and is_numeric_index_ops(right):
+ left = transform_boolean_operand_to_numeric(left, right.spark.data_type)
+ return numpy_column_op(floordiv)(left, right)
+ else:
+ raise TypeError(
+ "Floor division can not be applied to %s and the given type." % self.pretty_name)
+
+ def mod(self, left, right):
+ def mod(left, right):
+ return ((left % right) + right) % right
+
+ if isinstance(right, numbers.Number):
+ left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
+ return column_op(mod)(left, right)
+ elif isinstance(right, IndexOpsMixin) and is_numeric_index_ops(right):
+ left = transform_boolean_operand_to_numeric(left, right.spark.data_type)
+ return column_op(mod)(left, right)
+ else:
+ raise TypeError(
+ "Modulo can not be applied to %s and the given type." % self.pretty_name)
+
+ def pow(self, left, right):
+ def pow_func(left, right):
+ return F.when(left == 1, left).otherwise(Column.__pow__(left, right))
+
+ if isinstance(right, numbers.Number):
+ left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
+ return column_op(pow_func)(left, right)
+ elif isinstance(right, IndexOpsMixin) and is_numeric_index_ops(right):
+ left = transform_boolean_operand_to_numeric(left, right.spark.data_type)
+ return column_op(pow_func)(left, right)
+
+ else:
+ raise TypeError(
+ "Exponentiation can not be applied to %s and the given type." % self.pretty_name)
+
+ def radd(self, left, right=None):
+ if isinstance(right, numbers.Number):
+ left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
+ return column_op(Column.__radd__)(left, right)
+ else:
+ raise TypeError(
+ "Addition can not be applied to %s and the given type." % self.pretty_name)
+
+ def rsub(self, left, right=None):
Review comment:
We don't need `=None`?
##########
File path: python/pyspark/pandas/tests/data_type_ops/test_boolean_ops.py
##########
@@ -33,107 +35,148 @@ def pser(self):
def psser(self):
return ps.from_pandas(self.pser)
+ @property
+ def float_pser(self):
+ return pd.Series([1, 2, 3], dtype=float)
+
+ @property
+ def float_psser(self):
+ return ps.from_pandas(self.float_pser)
+
def test_add(self):
- self.assertRaises(TypeError, lambda: self.psser + 1)
- self.assertRaises(TypeError, lambda: self.psser + 0.1)
+ self.assert_eq(self.pser + 1, self.psser + 1)
+ self.assert_eq(self.pser + 0.1, self.psser + 0.1)
Review comment:
What if:
```py
psser = self.psser
psser + psser.astype(int)
psser + psser
```
##########
File path: python/pyspark/pandas/tests/data_type_ops/test_num_ops.py
##########
@@ -30,6 +34,14 @@ class NumOpsTest(PandasOnSparkTestCase, TestCasesUtils):
returns float32.
The underlying reason is the respective Spark operations return DoubleType always.
"""
+ @property
+ def float_pser(self):
+ return pd.Series([1, 2, 3], dtype=float)
+
+ @property
+ def float_psser(self):
+ return ps.from_pandas(self.float_pser)
+
def test_add(self):
for pser, psser in self.numeric_pser_psser_pairs:
self.assert_eq(pser + pser, psser + psser)
Review comment:
What if:
```py
psser + True
psser + psser.astype(bool)
```
##########
File path: python/pyspark/pandas/tests/data_type_ops/test_date_ops.py
##########
@@ -60,7 +61,9 @@ def test_sub(self):
with option_context("compute.ops_on_diff_frames", True):
for pser, psser in self.pser_psser_pairs:
if isinstance(psser.spark.data_type, DateType):
- self.assert_eq((self.pser - pser).dt.days, (self.psser - psser).sort_index())
+ if LooseVersion(pd.__version__) >= LooseVersion("0.24.2"):
+ self.assert_eq(
+ (self.pser - pser).dt.days, (self.psser - psser).sort_index())
Review comment:
btw, this is not related to this PR?
##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +41,183 @@ class BooleanOps(DataTypeOps):
@property
def pretty_name(self) -> str:
return 'booleans'
+
+ def add(self, left, right):
+ if isinstance(right, numbers.Number):
+ left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
Review comment:
`left.spark.transform` instead of `apply`?
##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +41,183 @@ class BooleanOps(DataTypeOps):
@property
def pretty_name(self) -> str:
return 'booleans'
+
+ def add(self, left, right):
Review comment:
Could you add type annotations?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846226896
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43333/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846317781
**[Test build #138818 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138818/testReport)** for PR 32611 at commit [`925e34b`](https://github.com/apache/spark/commit/925e34b3bffb6496d62425fa08d36177589cb17d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847421527
**[Test build #138892 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138892/testReport)** for PR 32611 at commit [`a608e5a`](https://github.com/apache/spark/commit/a608e5a57ad27236b4e412fd8e4d440937c26e61).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [WIP][SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845505274
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138766/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845618630
**[Test build #138785 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138785/testReport)** for PR 32611 at commit [`3fef44b`](https://github.com/apache/spark/commit/3fef44bea9b90463b01eda946939faddd88740a5).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845511899
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138768/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845535666
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138769/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845555582
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43291/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r640957039
##########
File path: python/pyspark/pandas/tests/test_dataframe.py
##########
@@ -121,7 +121,7 @@ def test_extension_dtypes(self):
psdf = ps.from_pandas(pdf)
self._check_extension(psdf, pdf)
- self._check_extension(psdf + F.lit(1).cast("byte"), pdf + 1)
+ # self._check_extension(psdf + F.lit(1).cast("byte"), pdf + 1)
Review comment:
@ueshin Shall we disable the test case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849218408
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43510/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847419100
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849972655
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43551/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849933905
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139033/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r639177972
##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +39,144 @@ class BooleanOps(DataTypeOps):
@property
def pretty_name(self) -> str:
return 'booleans'
+
+ def add(self, left, right) -> Union["Series", "Index"]:
+ if not is_valid_operand_for_numeric_arithmetic(right, allow_bool=False):
+ raise TypeError(
+ "Addition can not be applied to %s and the given type." % self.pretty_name)
+
+ if isinstance(right, numbers.Number) and not isinstance(right, bool):
+ left = left.spark.transform(lambda scol: scol.cast(as_spark_type(type(right))))
+ return left + right
+ else:
+ assert isinstance(right, IndexOpsMixin)
Review comment:
I see what you mean. I don't find an existing code path for `right` to be `spark.Column` for `boolean_ops` though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847350100
**[Test build #138888 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138888/testReport)** for PR 32611 at commit [`99e66a1`](https://github.com/apache/spark/commit/99e66a18a07f0ba69a0c036b36374dd8be74ba49).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r638317052
##########
File path: python/pyspark/pandas/data_type_ops/num_ops.py
##########
@@ -40,16 +40,19 @@
from pyspark.pandas.series import Series # noqa: F401 (SPARK-34943)
-def is_valid_operand_for_numeric_arithmetic(operand: Any) -> bool:
+def is_valid_operand_for_numeric_arithmetic(
+ operand: Any,
+ allow_bool_index_ops: bool = True
+) -> bool:
Review comment:
Good idea, `allow_bool` is set as a keyword-only argument.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r643361468
##########
File path: python/pyspark/pandas/tests/data_type_ops/test_num_ops.py
##########
@@ -30,6 +34,14 @@ class NumOpsTest(PandasOnSparkTestCase, TestCasesUtils):
returns float32.
The underlying reason is the respective Spark operations return DoubleType always.
"""
+ @property
+ def float_pser(self):
+ return pd.Series([1, 2, 3], dtype=float)
+
+ @property
+ def float_psser(self):
+ return ps.from_pandas(self.float_pser)
+
def test_add(self):
for pser, psser in self.numeric_pser_psser_pairs:
self.assert_eq(pser + pser, psser + psser)
Review comment:
https://issues.apache.org/jira/browse/SPARK-35601 is created for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847390918
**[Test build #138889 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138889/testReport)** for PR 32611 at commit [`7dac0d1`](https://github.com/apache/spark/commit/7dac0d1502f9265b118ed54dae4bdd69aadd6d32).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850542195
**[Test build #139066 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139066/testReport)** for PR 32611 at commit [`7b6d6a6`](https://github.com/apache/spark/commit/7b6d6a6986d9401692911ef7512ab260328dc115).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850707017
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139073/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849164895
**[Test build #138990 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138990/testReport)** for PR 32611 at commit [`98a5c8f`](https://github.com/apache/spark/commit/98a5c8ff8bec0dcfcb386fd970fedf55384f1571).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845618215
**[Test build #138785 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138785/testReport)** for PR 32611 at commit [`3fef44b`](https://github.com/apache/spark/commit/3fef44bea9b90463b01eda946939faddd88740a5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845702336
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43311/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847346303
**[Test build #138885 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138885/testReport)** for PR 32611 at commit [`5e2f911`](https://github.com/apache/spark/commit/5e2f91152dc0ad45c7155d94e4b87e82523ed04b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847466985
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43417/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847403018
**[Test build #138889 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138889/testReport)** for PR 32611 at commit [`7dac0d1`](https://github.com/apache/spark/commit/7dac0d1502f9265b118ed54dae4bdd69aadd6d32).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847270458
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43402/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-848024627
**[Test build #138936 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138936/testReport)** for PR 32611 at commit [`9a8f0b9`](https://github.com/apache/spark/commit/9a8f0b9a693281b5c66509c40e97a9525a216835).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846328527
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43340/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847390479
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138888/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849164895
**[Test build #138990 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138990/testReport)** for PR 32611 at commit [`98a5c8f`](https://github.com/apache/spark/commit/98a5c8ff8bec0dcfcb386fd970fedf55384f1571).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847242145
**[Test build #138882 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138882/testReport)** for PR 32611 at commit [`43a2972`](https://github.com/apache/spark/commit/43a29721e982df716a874286347e8b93492e1fd7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845509991
**[Test build #138768 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138768/testReport)** for PR 32611 at commit [`6c5a5cf`](https://github.com/apache/spark/commit/6c5a5cf431e0e23b67fdcbee578563dffb366947).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r637298978
##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +41,183 @@ class BooleanOps(DataTypeOps):
@property
def pretty_name(self) -> str:
return 'booleans'
+
+ def add(self, left, right):
+ if isinstance(right, numbers.Number):
+ left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
Review comment:
Cool! Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845652569
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43308/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-848070211
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138936/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r640174525
##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +39,144 @@ class BooleanOps(DataTypeOps):
@property
def pretty_name(self) -> str:
return 'booleans'
+
+ def add(self, left, right) -> Union["Series", "Index"]:
+ if not is_valid_operand_for_numeric_arithmetic(right, allow_bool=False):
+ raise TypeError(
+ "Addition can not be applied to %s and the given type." % self.pretty_name)
+
+ if isinstance(right, numbers.Number) and not isinstance(right, bool):
+ left = left.spark.transform(lambda scol: scol.cast(as_spark_type(type(right))))
+ return left + right
+ else:
+ assert isinstance(right, IndexOpsMixin)
Review comment:
Shall we disable `spark.Column` as operands for other Ops class in a separate PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [WIP][SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845470209
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43285/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845511899
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138768/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846111969
**[Test build #138811 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138811/testReport)** for PR 32611 at commit [`bd63058`](https://github.com/apache/spark/commit/bd630589cc6d686315053e6024cc61f4fb396bfd).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r638314268
##########
File path: python/pyspark/pandas/data_type_ops/num_ops.py
##########
@@ -40,16 +40,19 @@
from pyspark.pandas.series import Series # noqa: F401 (SPARK-34943)
-def is_valid_operand_for_numeric_arithmetic(operand: Any) -> bool:
+def is_valid_operand_for_numeric_arithmetic(
+ operand: Any,
+ allow_bool_index_ops: bool = True
+) -> bool:
Review comment:
Btw, the argument name should be like `allow_bool` or something since we will want to use it for bool literal as an operand in a following PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r638212053
##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -15,7 +15,27 @@
# limitations under the License.
#
-from pyspark.pandas.data_type_ops.base import DataTypeOps
+import numbers
+from typing import TYPE_CHECKING, Union
+
+import numpy as np
+from pandas.api.types import CategoricalDtype
+
+from pyspark.pandas.base import column_op, IndexOpsMixin, numpy_column_op
+from pyspark.pandas.data_type_ops.base import DataTypeOps, transform_boolean_operand_to_numeric
+from pyspark.pandas.typedef.typehints import as_spark_type
+from pyspark.sql import Column, functions as F
+from pyspark.sql.types import NumericType
+
+if TYPE_CHECKING:
+ from pyspark.pandas.indexes import Index # noqa: F401 (SPARK-34943)
+ from pyspark.pandas.series import Series # noqa: F401 (SPARK-34943)
+
+
+def is_numeric_index_ops(index_ops: IndexOpsMixin) -> bool:
+ """Check if the given index_ops is numeric IndexOpsMixin."""
+ return isinstance(index_ops.spark.data_type, NumericType) and (
+ not isinstance(index_ops.dtype, CategoricalDtype))
Review comment:
Sounds good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846080849
**[Test build #138809 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138809/testReport)** for PR 32611 at commit [`a123db8`](https://github.com/apache/spark/commit/a123db80c882b806c6c60fb56e83b8b5253e0b63).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845707439
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43311/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845557970
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43292/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849193160
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43509/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849188150
**[Test build #138991 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138991/testReport)** for PR 32611 at commit [`5905012`](https://github.com/apache/spark/commit/5905012338a16ba500094b9cae33bb783e86faff).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847310471
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43404/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-848110079
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43457/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845540848
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43291/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847417170
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43411/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847442495
**[Test build #138895 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138895/testReport)** for PR 32611 at commit [`203f90a`](https://github.com/apache/spark/commit/203f90aab81cecf2591f2fe2aa4b5466b2e486ae).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849211282
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138991/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r638166139
##########
File path: python/pyspark/pandas/tests/data_type_ops/test_boolean_ops.py
##########
@@ -33,107 +35,148 @@ def pser(self):
def psser(self):
return ps.from_pandas(self.pser)
+ @property
+ def float_pser(self):
+ return pd.Series([1, 2, 3], dtype=float)
+
+ @property
+ def float_psser(self):
+ return ps.from_pandas(self.float_pser)
+
def test_add(self):
- self.assertRaises(TypeError, lambda: self.psser + 1)
- self.assertRaises(TypeError, lambda: self.psser + 0.1)
+ self.assert_eq(self.pser + 1, self.psser + 1)
+ self.assert_eq(self.pser + 0.1, self.psser + 0.1)
with option_context("compute.ops_on_diff_frames", True):
- for psser in self.pssers:
+ for pser, psser in self.numeric_pser_psser_pairs:
+ self.assert_eq(self.pser + pser, (self.psser + psser).sort_index())
+
+ for psser in self.non_numeric_pssers.values():
self.assertRaises(TypeError, lambda: self.psser + psser)
def test_sub(self):
- self.assertRaises(TypeError, lambda: self.psser - 1)
- self.assertRaises(TypeError, lambda: self.psser - 0.1)
+ self.assert_eq(self.pser - 1, self.psser - 1)
+ self.assert_eq(self.pser - 0.1, self.psser - 0.1)
with option_context("compute.ops_on_diff_frames", True):
- for psser in self.pssers:
+ for pser, psser in self.numeric_pser_psser_pairs:
+ self.assert_eq(self.pser - pser, (self.psser - psser).sort_index())
+
+ for psser in self.non_numeric_pssers.values():
self.assertRaises(TypeError, lambda: self.psser - psser)
def test_mul(self):
- self.assertRaises(TypeError, lambda: self.psser * 1)
- self.assertRaises(TypeError, lambda: self.psser * 0.1)
+ self.assert_eq(self.pser * 1, self.psser * 1)
+ self.assert_eq(self.pser * 0.1, self.psser * 0.1)
with option_context("compute.ops_on_diff_frames", True):
- for psser in self.pssers:
+ for pser, psser in self.numeric_pser_psser_pairs:
+ self.assert_eq(self.pser * pser, (self.psser * psser).sort_index())
+
+ for psser in self.non_numeric_pssers.values():
self.assertRaises(TypeError, lambda: self.psser * psser)
def test_truediv(self):
- self.assertRaises(TypeError, lambda: self.psser / 1)
- self.assertRaises(TypeError, lambda: self.psser / 0.1)
+ self.assert_eq(self.pser / 1, self.psser / 1)
+ self.assert_eq(self.pser / 0.1, self.psser / 0.1)
with option_context("compute.ops_on_diff_frames", True):
- for psser in self.pssers:
+ self.assert_eq(
+ self.pser / self.float_pser, (self.psser / self.float_psser).sort_index())
+
+ for psser in self.non_numeric_pssers.values():
self.assertRaises(TypeError, lambda: self.psser / psser)
def test_floordiv(self):
- self.assertRaises(TypeError, lambda: self.psser // 1)
- self.assertRaises(TypeError, lambda: self.psser // 0.1)
+ # float is always returned in pandas-on-Spark
+ self.assert_eq((self.pser // 1).astype("float"), self.psser // 1)
+ # in pandas, 1 // 0.1 = 9.0; in pandas-on-Spark, 1 // 0.1 = 10.0
+ # self.assert_eq(self.pser // 0.1, self.psser // 0.1)
with option_context("compute.ops_on_diff_frames", True):
- for psser in self.pssers:
+ self.assert_eq(
+ self.pser // self.float_pser, (self.psser // self.float_psser).sort_index()
+ )
+
+ for psser in self.non_numeric_pssers.values():
self.assertRaises(TypeError, lambda: self.psser // psser)
def test_mod(self):
- self.assertRaises(TypeError, lambda: self.psser % 1)
- self.assertRaises(TypeError, lambda: self.psser % 0.1)
+ self.assert_eq(self.pser % 1, self.psser % 1)
+ self.assert_eq(self.pser % 0.1, self.psser % 0.1)
with option_context("compute.ops_on_diff_frames", True):
- for psser in self.pssers:
+ for pser, psser in self.numeric_pser_psser_pairs:
+ self.assert_eq(self.pser % pser, (self.psser % psser).sort_index())
+
+ for psser in self.non_numeric_pssers.values():
self.assertRaises(TypeError, lambda: self.psser % psser)
def test_pow(self):
- self.assertRaises(TypeError, lambda: self.psser ** 1)
- self.assertRaises(TypeError, lambda: self.psser ** 0.1)
+ # float is always returned in pandas-on-Spark
+ self.assert_eq((self.pser ** 1).astype("float"), self.psser ** 1)
+ self.assert_eq(self.pser ** 0.1, self.psser ** 0.1)
with option_context("compute.ops_on_diff_frames", True):
- for psser in self.pssers:
+ self.assert_eq(
+ self.pser ** self.float_pser, (self.psser ** self.float_psser).sort_index()
+ )
+
+ for psser in self.non_numeric_pssers.values():
self.assertRaises(TypeError, lambda: self.psser ** psser)
def test_radd(self):
- self.assertRaises(TypeError, lambda: 1 + self.psser)
- self.assertRaises(TypeError, lambda: 0.1 + self.psser)
+ self.assert_eq(1 + self.pser, 1 + self.psser)
+ self.assert_eq(0.1 + self.pser, 0.1 + self.psser)
self.assertRaises(TypeError, lambda: "x" + self.psser)
self.assertRaises(TypeError, lambda: datetime.date(1994, 1, 1) + self.psser)
self.assertRaises(TypeError, lambda: datetime.datetime(1994, 1, 1) + self.psser)
def test_rsub(self):
- self.assertRaises(TypeError, lambda: 1 - self.psser)
- self.assertRaises(TypeError, lambda: 0.1 - self.psser)
+ self.assert_eq(1 - self.pser, 1 - self.psser)
+ self.assert_eq(0.1 - self.pser, 0.1 - self.psser)
self.assertRaises(TypeError, lambda: "x" - self.psser)
self.assertRaises(TypeError, lambda: datetime.date(1994, 1, 1) - self.psser)
self.assertRaises(TypeError, lambda: datetime.datetime(1994, 1, 1) - self.psser)
def test_rmul(self):
- self.assertRaises(TypeError, lambda: 1 * self.psser)
- self.assertRaises(TypeError, lambda: 0.1 * self.psser)
+ self.assert_eq(1 * self.pser, 1 * self.psser)
+ self.assert_eq(0.1 * self.pser, 0.1 * self.psser)
self.assertRaises(TypeError, lambda: "x" * self.psser)
self.assertRaises(TypeError, lambda: datetime.date(1994, 1, 1) * self.psser)
self.assertRaises(TypeError, lambda: datetime.datetime(1994, 1, 1) * self.psser)
def test_rtruediv(self):
- self.assertRaises(TypeError, lambda: 1 / self.psser)
- self.assertRaises(TypeError, lambda: 0.1 / self.psser)
+ self.assert_eq(1 / self.pser, 1 / self.psser)
+ self.assert_eq(0.1 / self.pser, 0.1 / self.psser)
self.assertRaises(TypeError, lambda: "x" / self.psser)
self.assertRaises(TypeError, lambda: datetime.date(1994, 1, 1) / self.psser)
self.assertRaises(TypeError, lambda: datetime.datetime(1994, 1, 1) / self.psser)
def test_rfloordiv(self):
- self.assertRaises(TypeError, lambda: 1 // self.psser)
- self.assertRaises(TypeError, lambda: 0.1 // self.psser)
+ if LooseVersion(pd.__version__) >= LooseVersion("0.25.3"):
+ self.assert_eq(1 // self.pser, 1 // self.psser)
+ self.assert_eq(0.1 // self.pser, 0.1 // self.psser)
Review comment:
`else` case is added.
##########
File path: python/pyspark/pandas/tests/data_type_ops/test_num_ops.py
##########
@@ -101,7 +133,13 @@ def test_floordiv(self):
self.assertRaises(TypeError, lambda: psser // self.non_numeric_pssers["date"])
self.assertRaises(
TypeError, lambda: psser // self.non_numeric_pssers["categorical"])
- self.assertRaises(TypeError, lambda: psser // self.non_numeric_pssers["bool"])
+ if LooseVersion(pd.__version__) >= LooseVersion("0.25.3"):
+ self.assert_eq(
+ (self.float_psser // self.non_numeric_pssers["bool"]).sort_index(),
+ self.float_pser // self.non_numeric_psers["bool"],
+ )
Review comment:
`else` case is added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r638336163
##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +38,136 @@ class BooleanOps(DataTypeOps):
@property
def pretty_name(self) -> str:
return 'booleans'
+
+ def add(self, left, right) -> Union["Series", "Index"]:
+ if isinstance(right, numbers.Number):
Review comment:
Makes sense, disabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r638131359
##########
File path: python/pyspark/pandas/tests/data_type_ops/test_num_ops.py
##########
@@ -30,6 +34,14 @@ class NumOpsTest(PandasOnSparkTestCase, TestCasesUtils):
returns float32.
The underlying reason is the respective Spark operations return DoubleType always.
"""
+ @property
+ def float_pser(self):
+ return pd.Series([1, 2, 3], dtype=float)
+
+ @property
+ def float_psser(self):
+ return ps.from_pandas(self.float_pser)
+
def test_add(self):
for pser, psser in self.numeric_pser_psser_pairs:
self.assert_eq(pser + pser, psser + psser)
Review comment:
Arithmetic operations between `psser` and `psser.astype(bool)` are tested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r637301587
##########
File path: python/pyspark/pandas/tests/data_type_ops/test_date_ops.py
##########
@@ -60,7 +61,9 @@ def test_sub(self):
with option_context("compute.ops_on_diff_frames", True):
for pser, psser in self.pser_psser_pairs:
if isinstance(psser.spark.data_type, DateType):
- self.assert_eq((self.pser - pser).dt.days, (self.psser - psser).sort_index())
+ if LooseVersion(pd.__version__) >= LooseVersion("0.24.2"):
+ self.assert_eq(
+ (self.pser - pser).dt.days, (self.psser - psser).sort_index())
Review comment:
It raises weird TypeError for pandas in the `else` case, as https://github.com/databricks/koalas/runs/2568962950.
No, the change is not related to this PR. Let me remove it for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846144301
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845652569
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43308/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin closed pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
ueshin closed pull request #32611:
URL: https://github.com/apache/spark/pull/32611
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845556717
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r638132565
##########
File path: python/pyspark/pandas/tests/data_type_ops/test_num_ops.py
##########
@@ -42,7 +54,12 @@ def test_add(self):
self.assertRaises(TypeError, lambda: psser + self.non_numeric_pssers["datetime"])
self.assertRaises(TypeError, lambda: psser + self.non_numeric_pssers["date"])
self.assertRaises(TypeError, lambda: psser + self.non_numeric_pssers["categorical"])
- self.assertRaises(TypeError, lambda: psser + self.non_numeric_pssers["bool"])
+ self.assert_eq(
+ (psser + self.non_numeric_pssers["bool"]).sort_index(),
+ pser + self.non_numeric_psers["bool"],
+ )
+
+ self.assertRaises(TypeError, lambda: self.float_psser + True)
Review comment:
Goot catch! It is out of `with option_context` now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846142367
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43333/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847350100
**[Test build #138888 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138888/testReport)** for PR 32611 at commit [`99e66a1`](https://github.com/apache/spark/commit/99e66a18a07f0ba69a0c036b36374dd8be74ba49).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847428962
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43411/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847346980
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138885/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849187308
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138990/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847230904
**[Test build #138880 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138880/testReport)** for PR 32611 at commit [`b4e2bc7`](https://github.com/apache/spark/commit/b4e2bc7689cdac95e4358e48754f8cdb4cf313fb).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847439897
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43414/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849218408
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43510/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r639195854
##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +39,144 @@ class BooleanOps(DataTypeOps):
@property
def pretty_name(self) -> str:
return 'booleans'
+
+ def add(self, left, right) -> Union["Series", "Index"]:
+ if not is_valid_operand_for_numeric_arithmetic(right, allow_bool=False):
+ raise TypeError(
+ "Addition can not be applied to %s and the given type." % self.pretty_name)
+
+ if isinstance(right, numbers.Number) and not isinstance(right, bool):
+ left = left.spark.transform(lambda scol: scol.cast(as_spark_type(type(right))))
+ return left + right
+ else:
+ assert isinstance(right, IndexOpsMixin)
Review comment:
You mentioned that `spark.Column` is allowed to be operands (https://github.com/apache/spark/pull/32611#discussion_r636607352), and if so, we should handle it properly.
> And mypy likes this. Would you prefer to use cast instead?
What do you mean?
##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +39,144 @@ class BooleanOps(DataTypeOps):
@property
def pretty_name(self) -> str:
return 'booleans'
+
+ def add(self, left, right) -> Union["Series", "Index"]:
+ if not is_valid_operand_for_numeric_arithmetic(right, allow_bool=False):
+ raise TypeError(
+ "Addition can not be applied to %s and the given type." % self.pretty_name)
+
+ if isinstance(right, numbers.Number) and not isinstance(right, bool):
+ left = left.spark.transform(lambda scol: scol.cast(as_spark_type(type(right))))
+ return left + right
+ else:
+ assert isinstance(right, IndexOpsMixin)
Review comment:
You mentioned that `spark.Column` is allowed to be operands (https://github.com/apache/spark/pull/32611#discussion_r636607352), and if so, we should handle it properly.
> And mypy likes this. Would you prefer to use cast instead?
What do you mean?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850711760
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43594/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [WIP][SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845431129
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138763/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850542195
**[Test build #139066 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139066/testReport)** for PR 32611 at commit [`7b6d6a6`](https://github.com/apache/spark/commit/7b6d6a6986d9401692911ef7512ab260328dc115).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847466995
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43417/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r637338469
##########
File path: python/pyspark/pandas/data_type_ops/num_ops.py
##########
@@ -16,31 +16,46 @@
#
import numbers
-from typing import TYPE_CHECKING, Union
+from typing import Any, TYPE_CHECKING, Union
import numpy as np
from pandas.api.types import CategoricalDtype
from pyspark.sql import Column, functions as F
from pyspark.sql.types import (
+ BooleanType,
NumericType,
StringType,
TimestampType,
)
from pyspark.pandas.base import column_op, IndexOpsMixin, numpy_column_op
-from pyspark.pandas.data_type_ops.base import DataTypeOps
+from pyspark.pandas.data_type_ops.base import DataTypeOps, transform_boolean_operand_to_numeric
from pyspark.pandas.spark import functions as SF
+from pyspark.sql.column import Column
+
if TYPE_CHECKING:
from pyspark.pandas.indexes import Index # noqa: F401 (SPARK-34943)
from pyspark.pandas.series import Series # noqa: F401 (SPARK-34943)
+def is_valid_operand_for_numeric_arithmetic(operand: Any) -> bool:
+ """Check whether the operand is valid for arithmetic operations against numerics."""
+ if isinstance(operand, numbers.Number):
+ return True
+ elif isinstance(operand, IndexOpsMixin):
+ if isinstance(operand.dtype, CategoricalDtype):
+ return False
+ else:
+ return isinstance(operand.spark.data_type, NumericType) or isinstance(
+ operand.spark.data_type, BooleanType)
+ else:
+ return isinstance(operand, Column)
Review comment:
Do we currently allow it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r641682948
##########
File path: python/pyspark/pandas/tests/test_dataframe.py
##########
@@ -121,7 +121,7 @@ def test_extension_dtypes(self):
psdf = ps.from_pandas(pdf)
self._check_extension(psdf, pdf)
- self._check_extension(psdf + F.lit(1).cast("byte"), pdf + 1)
+ # self._check_extension(psdf + F.lit(1).cast("byte"), pdf + 1)
Review comment:
Cool.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845577570
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43289/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32611: [WIP][SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845389964
**[Test build #138763 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138763/testReport)** for PR 32611 at commit [`d44a3b4`](https://github.com/apache/spark/commit/d44a3b4512613f071c2bd48d3aa8cd06be78a07d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845615298
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43288/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846226896
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43333/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [WIP][SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845448415
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43285/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846144303
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847431065
**[Test build #138892 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138892/testReport)** for PR 32611 at commit [`a608e5a`](https://github.com/apache/spark/commit/a608e5a57ad27236b4e412fd8e4d440937c26e61).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846111969
**[Test build #138811 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138811/testReport)** for PR 32611 at commit [`bd63058`](https://github.com/apache/spark/commit/bd630589cc6d686315053e6024cc61f4fb396bfd).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846326242
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138818/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847390479
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138888/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847441076
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-848048563
**[Test build #138936 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138936/testReport)** for PR 32611 at commit [`9a8f0b9`](https://github.com/apache/spark/commit/9a8f0b9a693281b5c66509c40e97a9525a216835).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845535666
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138769/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847242145
**[Test build #138882 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138882/testReport)** for PR 32611 at commit [`43a2972`](https://github.com/apache/spark/commit/43a29721e982df716a874286347e8b93492e1fd7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847209831
**[Test build #138880 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138880/testReport)** for PR 32611 at commit [`b4e2bc7`](https://github.com/apache/spark/commit/b4e2bc7689cdac95e4358e48754f8cdb4cf313fb).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r637328527
##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +46,183 @@ class BooleanOps(DataTypeOps):
@property
def pretty_name(self) -> str:
return 'booleans'
+
+ def add(self, left, right) -> Union["Series", "Index"]:
+ if isinstance(right, numbers.Number):
+ left = left.spark.transform(lambda scol: scol.cast(as_spark_type(type(right))))
+ return column_op(Column.__add__)(left, right)
Review comment:
I guess `return left + right` should be fine. The `DataTypeOps` related to `left` after converted should know how to handle it better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846324042
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43340/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850566522
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43587/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r638140558
##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -15,7 +15,27 @@
# limitations under the License.
#
-from pyspark.pandas.data_type_ops.base import DataTypeOps
+import numbers
+from typing import TYPE_CHECKING, Union
+
+import numpy as np
+from pandas.api.types import CategoricalDtype
+
+from pyspark.pandas.base import column_op, IndexOpsMixin, numpy_column_op
+from pyspark.pandas.data_type_ops.base import DataTypeOps, transform_boolean_operand_to_numeric
+from pyspark.pandas.typedef.typehints import as_spark_type
+from pyspark.sql import Column, functions as F
+from pyspark.sql.types import NumericType
+
+if TYPE_CHECKING:
+ from pyspark.pandas.indexes import Index # noqa: F401 (SPARK-34943)
+ from pyspark.pandas.series import Series # noqa: F401 (SPARK-34943)
+
+
+def is_numeric_index_ops(index_ops: IndexOpsMixin) -> bool:
+ """Check if the given index_ops is numeric IndexOpsMixin."""
+ return isinstance(index_ops.spark.data_type, NumericType) and (
+ not isinstance(index_ops.dtype, CategoricalDtype))
Review comment:
`is_valid_operand_for_numeric_arithmetic` returns True for `BooleanType`, whereas `is_numeric_index_ops ` returns False for `BooleanType`.
We may also pass in a boolean flag deciding how to deal with `BooleanType`. How do you think about this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845618643
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138785/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r637311351
##########
File path: python/pyspark/pandas/tests/data_type_ops/test_num_ops.py
##########
@@ -30,6 +34,14 @@ class NumOpsTest(PandasOnSparkTestCase, TestCasesUtils):
returns float32.
The underlying reason is the respective Spark operations return DoubleType always.
"""
+ @property
+ def float_pser(self):
+ return pd.Series([1, 2, 3], dtype=float)
+
+ @property
+ def float_psser(self):
+ return ps.from_pandas(self.float_pser)
+
def test_add(self):
for pser, psser in self.numeric_pser_psser_pairs:
self.assert_eq(pser + pser, psser + psser)
Review comment:
Isn't this PR supposed to support it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-848070211
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138936/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845618215
**[Test build #138785 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138785/testReport)** for PR 32611 at commit [`3fef44b`](https://github.com/apache/spark/commit/3fef44bea9b90463b01eda946939faddd88740a5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-848110079
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43457/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847277529
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846326242
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138818/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849174414
**[Test build #138990 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138990/testReport)** for PR 32611 at commit [`98a5c8f`](https://github.com/apache/spark/commit/98a5c8ff8bec0dcfcb386fd970fedf55384f1571).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847417614
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [WIP][SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845389964
**[Test build #138763 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138763/testReport)** for PR 32611 at commit [`d44a3b4`](https://github.com/apache/spark/commit/d44a3b4512613f071c2bd48d3aa8cd06be78a07d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r639177972
##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +39,144 @@ class BooleanOps(DataTypeOps):
@property
def pretty_name(self) -> str:
return 'booleans'
+
+ def add(self, left, right) -> Union["Series", "Index"]:
+ if not is_valid_operand_for_numeric_arithmetic(right, allow_bool=False):
+ raise TypeError(
+ "Addition can not be applied to %s and the given type." % self.pretty_name)
+
+ if isinstance(right, numbers.Number) and not isinstance(right, bool):
+ left = left.spark.transform(lambda scol: scol.cast(as_spark_type(type(right))))
+ return left + right
+ else:
+ assert isinstance(right, IndexOpsMixin)
Review comment:
I see what you mean. I don't find an existing code path for `right` to be `spark.Column` for `boolean_ops` though. And `mypy` likes this. Would you prefer to use `cast` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847384479
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43407/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847461342
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43414/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849933881
**[Test build #139033 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139033/testReport)** for PR 32611 at commit [`adfc5cc`](https://github.com/apache/spark/commit/adfc5cc6654a51105ef834525cdf1ebcdcd0a7bb).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850542883
**[Test build #139066 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139066/testReport)** for PR 32611 at commit [`7b6d6a6`](https://github.com/apache/spark/commit/7b6d6a6986d9401692911ef7512ab260328dc115).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845652182
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43308/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845543774
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43289/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845635027
**[Test build #138788 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138788/testReport)** for PR 32611 at commit [`e626c52`](https://github.com/apache/spark/commit/e626c52245521fdeb0ee46c68950605f87a987bd).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847277529
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847461342
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43414/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850703378
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43594/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847451920
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43414/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845535649
**[Test build #138769 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138769/testReport)** for PR 32611 at commit [`b0b2a3b`](https://github.com/apache/spark/commit/b0b2a3bb61680720bb2c1be86c87a2f8dfa847d9).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32611: [WIP][SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845504281
**[Test build #138766 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138766/testReport)** for PR 32611 at commit [`3f0fc27`](https://github.com/apache/spark/commit/3f0fc27ffe093ff6ef12c3f5214f6fa98c635860).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [WIP][SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845505274
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138766/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r638307070
##########
File path: python/pyspark/pandas/data_type_ops/num_ops.py
##########
@@ -40,16 +40,19 @@
from pyspark.pandas.series import Series # noqa: F401 (SPARK-34943)
-def is_valid_operand_for_numeric_arithmetic(operand: Any) -> bool:
+def is_valid_operand_for_numeric_arithmetic(
+ operand: Any,
+ allow_bool_index_ops: bool = True
+) -> bool:
Review comment:
Shall we make `allow_bool_index_ops` a keyword-only argument?
```py
def is_valid_operand_for_numeric_arithmetic(
operand: Any,
*,
allow_bool_index_ops: bool = True
) -> bool:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847310471
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43404/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850699633
**[Test build #139073 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139073/testReport)** for PR 32611 at commit [`db12fb1`](https://github.com/apache/spark/commit/db12fb1c854051b48caf6a253ef6dcb70694db51).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849933240
**[Test build #139033 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139033/testReport)** for PR 32611 at commit [`adfc5cc`](https://github.com/apache/spark/commit/adfc5cc6654a51105ef834525cdf1ebcdcd0a7bb).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847275778
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43404/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845575884
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43289/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845615298
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43288/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r637332388
##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -15,7 +15,27 @@
# limitations under the License.
#
-from pyspark.pandas.data_type_ops.base import DataTypeOps
+import numbers
+from typing import TYPE_CHECKING, Union
+
+import numpy as np
+from pandas.api.types import CategoricalDtype
+
+from pyspark.pandas.base import column_op, IndexOpsMixin, numpy_column_op
+from pyspark.pandas.data_type_ops.base import DataTypeOps, transform_boolean_operand_to_numeric
+from pyspark.pandas.typedef.typehints import as_spark_type
+from pyspark.sql import Column, functions as F
+from pyspark.sql.types import NumericType
+
+if TYPE_CHECKING:
+ from pyspark.pandas.indexes import Index # noqa: F401 (SPARK-34943)
+ from pyspark.pandas.series import Series # noqa: F401 (SPARK-34943)
+
+
+def is_numeric_index_ops(index_ops: IndexOpsMixin) -> bool:
+ """Check if the given index_ops is numeric IndexOpsMixin."""
+ return isinstance(index_ops.spark.data_type, NumericType) and (
+ not isinstance(index_ops.dtype, CategoricalDtype))
Review comment:
I guess we can reuse `is_valid_operand_for_numeric_arithmetic` in `num_ops.py`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849218398
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43510/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849211282
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138991/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845509991
**[Test build #138768 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138768/testReport)** for PR 32611 at commit [`6c5a5cf`](https://github.com/apache/spark/commit/6c5a5cf431e0e23b67fdcbee578563dffb366947).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850690763
**[Test build #139073 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139073/testReport)** for PR 32611 at commit [`db12fb1`](https://github.com/apache/spark/commit/db12fb1c854051b48caf6a253ef6dcb70694db51).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849193141
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43509/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846328532
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43340/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-848024627
**[Test build #138936 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138936/testReport)** for PR 32611 at commit [`9a8f0b9`](https://github.com/apache/spark/commit/9a8f0b9a693281b5c66509c40e97a9525a216835).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847397923
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43407/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [WIP][SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845419493
**[Test build #138763 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138763/testReport)** for PR 32611 at commit [`d44a3b4`](https://github.com/apache/spark/commit/d44a3b4512613f071c2bd48d3aa8cd06be78a07d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
ueshin commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-852331508
Thanks! merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846132387
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43332/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847346303
**[Test build #138885 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138885/testReport)** for PR 32611 at commit [`5e2f911`](https://github.com/apache/spark/commit/5e2f91152dc0ad45c7155d94e4b87e82523ed04b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846081872
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138809/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845585115
@xinrong-databricks can we show some examples in the PR description?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849933240
**[Test build #139033 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139033/testReport)** for PR 32611 at commit [`adfc5cc`](https://github.com/apache/spark/commit/adfc5cc6654a51105ef834525cdf1ebcdcd0a7bb).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850725471
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43594/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847390918
**[Test build #138889 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138889/testReport)** for PR 32611 at commit [`7dac0d1`](https://github.com/apache/spark/commit/7dac0d1502f9265b118ed54dae4bdd69aadd6d32).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846081226
CC @ueshin @HyukjinKwon @itholic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [WIP][SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845504281
**[Test build #138766 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138766/testReport)** for PR 32611 at commit [`3f0fc27`](https://github.com/apache/spark/commit/3f0fc27ffe093ff6ef12c3f5214f6fa98c635860).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846113323
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43332/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847442495
**[Test build #138895 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138895/testReport)** for PR 32611 at commit [`203f90a`](https://github.com/apache/spark/commit/203f90aab81cecf2591f2fe2aa4b5466b2e486ae).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r638127919
##########
File path: python/pyspark/pandas/tests/data_type_ops/test_boolean_ops.py
##########
@@ -33,107 +35,148 @@ def pser(self):
def psser(self):
return ps.from_pandas(self.pser)
+ @property
+ def float_pser(self):
+ return pd.Series([1, 2, 3], dtype=float)
+
+ @property
+ def float_psser(self):
+ return ps.from_pandas(self.float_pser)
+
def test_add(self):
- self.assertRaises(TypeError, lambda: self.psser + 1)
- self.assertRaises(TypeError, lambda: self.psser + 0.1)
+ self.assert_eq(self.pser + 1, self.psser + 1)
+ self.assert_eq(self.pser + 0.1, self.psser + 0.1)
Review comment:
Arithmetic operations between`psser` and `psser.astype(int)` are tested.
`psser + psser` will raise `TypeError`, while pandas treat it as `__and__`. A follow-up PR might be needed to support it. Would you prefer it to be implemented in this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847442932
**[Test build #138895 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138895/testReport)** for PR 32611 at commit [`203f90a`](https://github.com/apache/spark/commit/203f90aab81cecf2591f2fe2aa4b5466b2e486ae).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847391398
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43410/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850542913
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139066/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [WIP][SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845425762
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43285/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845618643
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138785/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847419100
**[Test build #138891 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138891/testReport)** for PR 32611 at commit [`7ea2876`](https://github.com/apache/spark/commit/7ea2876d0c5083ffa2eddbc634edab6e4a74d71a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846130825
**[Test build #138811 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138811/testReport)** for PR 32611 at commit [`bd63058`](https://github.com/apache/spark/commit/bd630589cc6d686315053e6024cc61f4fb396bfd).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845635027
**[Test build #138788 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138788/testReport)** for PR 32611 at commit [`e626c52`](https://github.com/apache/spark/commit/e626c52245521fdeb0ee46c68950605f87a987bd).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-848059344
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43457/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849205409
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43510/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845557983
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43292/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845534978
**[Test build #138769 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138769/testReport)** for PR 32611 at commit [`b0b2a3b`](https://github.com/apache/spark/commit/b0b2a3bb61680720bb2c1be86c87a2f8dfa847d9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845635353
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138788/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846317781
**[Test build #138818 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138818/testReport)** for PR 32611 at commit [`925e34b`](https://github.com/apache/spark/commit/925e34b3bffb6496d62425fa08d36177589cb17d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r640957750
##########
File path: python/pyspark/pandas/tests/test_dataframe.py
##########
@@ -121,7 +121,7 @@ def test_extension_dtypes(self):
psdf = ps.from_pandas(pdf)
self._check_extension(psdf, pdf)
- self._check_extension(psdf + F.lit(1).cast("byte"), pdf + 1)
+ # self._check_extension(psdf + F.lit(1).cast("byte"), pdf + 1)
Review comment:
Yes, shall we just remove it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847245881
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43402/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846328532
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43340/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845534978
**[Test build #138769 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138769/testReport)** for PR 32611 at commit [`b0b2a3b`](https://github.com/apache/spark/commit/b0b2a3bb61680720bb2c1be86c87a2f8dfa847d9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847365650
**[Test build #138888 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138888/testReport)** for PR 32611 at commit [`99e66a1`](https://github.com/apache/spark/commit/99e66a18a07f0ba69a0c036b36374dd8be74ba49).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847435504
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43413/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847240922
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138880/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845707439
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43311/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845556717
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43291/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846081872
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138809/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849985493
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43551/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845635337
**[Test build #138788 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138788/testReport)** for PR 32611 at commit [`e626c52`](https://github.com/apache/spark/commit/e626c52245521fdeb0ee46c68950605f87a987bd).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850707017
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139073/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845511826
**[Test build #138768 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138768/testReport)** for PR 32611 at commit [`6c5a5cf`](https://github.com/apache/spark/commit/6c5a5cf431e0e23b67fdcbee578563dffb366947).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850690763
**[Test build #139073 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139073/testReport)** for PR 32611 at commit [`db12fb1`](https://github.com/apache/spark/commit/db12fb1c854051b48caf6a253ef6dcb70694db51).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r638137564
##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +46,183 @@ class BooleanOps(DataTypeOps):
@property
def pretty_name(self) -> str:
return 'booleans'
+
+ def add(self, left, right) -> Union["Series", "Index"]:
+ if isinstance(right, numbers.Number):
+ left = left.spark.transform(lambda scol: scol.cast(as_spark_type(type(right))))
+ return column_op(Column.__add__)(left, right)
Review comment:
Makes sense, addition and other forward operations are applied this approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847441076
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846205753
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43333/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849187308
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138990/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847442945
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138895/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849188150
**[Test build #138991 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138991/testReport)** for PR 32611 at commit [`5905012`](https://github.com/apache/spark/commit/5905012338a16ba500094b9cae33bb783e86faff).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849957217
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43551/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r636607352
##########
File path: python/pyspark/pandas/data_type_ops/num_ops.py
##########
@@ -16,31 +16,46 @@
#
import numbers
-from typing import TYPE_CHECKING, Union
+from typing import Any, TYPE_CHECKING, Union
import numpy as np
from pandas.api.types import CategoricalDtype
from pyspark.sql import Column, functions as F
from pyspark.sql.types import (
+ BooleanType,
NumericType,
StringType,
TimestampType,
)
from pyspark.pandas.base import column_op, IndexOpsMixin, numpy_column_op
-from pyspark.pandas.data_type_ops.base import DataTypeOps
+from pyspark.pandas.data_type_ops.base import DataTypeOps, transform_boolean_operand_to_numeric
from pyspark.pandas.spark import functions as SF
+from pyspark.sql.column import Column
+
if TYPE_CHECKING:
from pyspark.pandas.indexes import Index # noqa: F401 (SPARK-34943)
from pyspark.pandas.series import Series # noqa: F401 (SPARK-34943)
+def is_valid_operand_for_numeric_arithmetic(operand: Any) -> bool:
+ """Check whether the operand is valid for arithmetic operations against numerics."""
+ if isinstance(operand, numbers.Number):
+ return True
+ elif isinstance(operand, IndexOpsMixin):
+ if isinstance(operand.dtype, CategoricalDtype):
+ return False
+ else:
+ return isinstance(operand.spark.data_type, NumericType) or isinstance(
+ operand.spark.data_type, BooleanType)
+ else:
+ return isinstance(operand, Column)
Review comment:
Returning True when the operand is a Column to cover the cases as below:
`left.astype("long") - F.lit(right).cast(as_spark_type("long"))` (when a datatime series - a datatime)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849933905
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139033/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850569990
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43587/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850569990
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43587/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847240922
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138880/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r637305932
##########
File path: python/pyspark/pandas/tests/data_type_ops/test_num_ops.py
##########
@@ -30,6 +34,14 @@ class NumOpsTest(PandasOnSparkTestCase, TestCasesUtils):
returns float32.
The underlying reason is the respective Spark operations return DoubleType always.
"""
+ @property
+ def float_pser(self):
+ return pd.Series([1, 2, 3], dtype=float)
+
+ @property
+ def float_psser(self):
+ return ps.from_pandas(self.float_pser)
+
def test_add(self):
for pser, psser in self.numeric_pser_psser_pairs:
self.assert_eq(pser + pser, psser + psser)
Review comment:
Arithmetic operations involving `bool` are not supported now. TypeErrors are introduced for these cases for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847417614
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845635353
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138788/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847442945
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138895/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r639154215
##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +39,144 @@ class BooleanOps(DataTypeOps):
@property
def pretty_name(self) -> str:
return 'booleans'
+
+ def add(self, left, right) -> Union["Series", "Index"]:
+ if not is_valid_operand_for_numeric_arithmetic(right, allow_bool=False):
+ raise TypeError(
+ "Addition can not be applied to %s and the given type." % self.pretty_name)
+
+ if isinstance(right, numbers.Number) and not isinstance(right, bool):
+ left = left.spark.transform(lambda scol: scol.cast(as_spark_type(type(right))))
+ return left + right
+ else:
+ assert isinstance(right, IndexOpsMixin)
Review comment:
I guess `right` of type `spark.Column` could fall into this path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847455951
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43417/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org