You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/05/20 18:30:49 UTC

[GitHub] [spark] xinrong-databricks opened a new pull request #32611: [WIP][SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

xinrong-databricks opened a new pull request #32611:
URL: https://github.com/apache/spark/pull/32611


   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
     2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
     4. Be sure to keep the PR description updated to reflect all changes.
     5. Please write your PR title to summarize what this PR proposes.
     6. If possible, provide a concise example to reproduce the issue for a faster review.
     7. If you want to add a new configuration, please read the guideline first for naming configurations in
        'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
     8. If you want to add or modify an error message, please read the guideline first:
        https://spark.apache.org/error-message-guidelines.html
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847346961


   **[Test build #138885 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138885/testReport)** for PR 32611 at commit [`5e2f911`](https://github.com/apache/spark/commit/5e2f91152dc0ad45c7155d94e4b87e82523ed04b).
    * This patch **fails Python style tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850725471


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43594/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r637299344



##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +41,183 @@ class BooleanOps(DataTypeOps):
     @property
     def pretty_name(self) -> str:
         return 'booleans'
+
+    def add(self, left, right):
+        if isinstance(right, numbers.Number):
+            left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
+            return column_op(Column.__add__)(left, right)
+        elif isinstance(right, IndexOpsMixin) and is_numeric_index_ops(right):
+            left = transform_boolean_operand_to_numeric(left, right.spark.data_type)
+            return column_op(Column.__add__)(left, right)
+        else:
+            raise TypeError(
+                "Addition can not be applied to %s and the given type." % self.pretty_name)
+
+    def sub(self, left, right):
+        if isinstance(right, numbers.Number):
+            left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
+            return column_op(Column.__sub__)(left, right)
+        elif isinstance(right, IndexOpsMixin) and is_numeric_index_ops(right):
+            left = transform_boolean_operand_to_numeric(left, right.spark.data_type)
+            return column_op(Column.__sub__)(left, right)
+        else:
+            raise TypeError(
+                "Subtraction can not be applied to %s and the given type." % self.pretty_name)
+
+    def mul(self, left, right):
+        if isinstance(right, numbers.Number):
+            left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
+            return column_op(Column.__mul__)(left, right)
+        elif isinstance(right, IndexOpsMixin) and is_numeric_index_ops(right):
+            left = transform_boolean_operand_to_numeric(left, right.spark.data_type)
+            return column_op(Column.__mul__)(left, right)
+        else:
+            raise TypeError(
+                "Multiplication can not be applied to %s and the given type." % self.pretty_name)
+
+    def truediv(self, left, right):
+        def truediv(left, right):
+            return F.when(F.lit(right != 0) | F.lit(right).isNull(), left.__div__(right)).otherwise(
+                F.when(F.lit(left == np.inf) | F.lit(left == -np.inf), left).otherwise(
+                    F.lit(np.inf).__div__(left)
+                )
+            )
+
+        if isinstance(right, numbers.Number):
+            left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
+            return numpy_column_op(truediv)(left, right)
+        elif isinstance(right, IndexOpsMixin) and is_numeric_index_ops(right):
+            left = transform_boolean_operand_to_numeric(left, right.spark.data_type)
+            return numpy_column_op(truediv)(left, right)
+        else:
+            raise TypeError(
+                "True division can not be applied to %s and the given type." % self.pretty_name)
+
+    def floordiv(self, left, right):
+        def floordiv(left, right):
+            return F.when(F.lit(right is np.nan), np.nan).otherwise(
+                F.when(
+                    F.lit(right != 0) | F.lit(right).isNull(), F.floor(left.__div__(right))
+                ).otherwise(
+                    F.when(F.lit(left == np.inf) | F.lit(left == -np.inf), left).otherwise(
+                        F.lit(np.inf).__div__(left)
+                    )
+                )
+            )
+
+        if isinstance(right, numbers.Number):
+            left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
+            return numpy_column_op(floordiv)(left, right)
+        elif isinstance(right, IndexOpsMixin) and is_numeric_index_ops(right):
+            left = transform_boolean_operand_to_numeric(left, right.spark.data_type)
+            return numpy_column_op(floordiv)(left, right)
+        else:
+            raise TypeError(
+                "Floor division can not be applied to %s and the given type." % self.pretty_name)
+
+    def mod(self, left, right):
+        def mod(left, right):
+            return ((left % right) + right) % right
+
+        if isinstance(right, numbers.Number):
+            left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
+            return column_op(mod)(left, right)
+        elif isinstance(right, IndexOpsMixin) and is_numeric_index_ops(right):
+            left = transform_boolean_operand_to_numeric(left, right.spark.data_type)
+            return column_op(mod)(left, right)
+        else:
+            raise TypeError(
+                "Modulo can not be applied to %s and the given type." % self.pretty_name)
+
+    def pow(self, left, right):
+        def pow_func(left, right):
+            return F.when(left == 1, left).otherwise(Column.__pow__(left, right))
+
+        if isinstance(right, numbers.Number):
+            left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
+            return column_op(pow_func)(left, right)
+        elif isinstance(right, IndexOpsMixin) and is_numeric_index_ops(right):
+            left = transform_boolean_operand_to_numeric(left, right.spark.data_type)
+            return column_op(pow_func)(left, right)
+
+        else:
+            raise TypeError(
+                "Exponentiation can not be applied to %s and the given type." % self.pretty_name)
+
+    def radd(self, left, right=None):
+        if isinstance(right, numbers.Number):
+            left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
+            return column_op(Column.__radd__)(left, right)
+        else:
+            raise TypeError(
+                "Addition can not be applied to %s and the given type." % self.pretty_name)
+
+    def rsub(self, left, right=None):

Review comment:
       `=None`s are removed.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849985493


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43551/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845577570


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43289/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849197264


   **[Test build #138991 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138991/testReport)** for PR 32611 at commit [`5905012`](https://github.com/apache/spark/commit/5905012338a16ba500094b9cae33bb783e86faff).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [WIP][SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845505250


   **[Test build #138766 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138766/testReport)** for PR 32611 at commit [`3f0fc27`](https://github.com/apache/spark/commit/3f0fc27ffe093ff6ef12c3f5214f6fa98c635860).
    * This patch **fails Python style tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847346980


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138885/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847263424


   **[Test build #138882 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138882/testReport)** for PR 32611 at commit [`43a2972`](https://github.com/apache/spark/commit/43a29721e982df716a874286347e8b93492e1fd7).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] ueshin commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r638330196



##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +38,136 @@ class BooleanOps(DataTypeOps):
     @property
     def pretty_name(self) -> str:
         return 'booleans'
+
+    def add(self, left, right) -> Union["Series", "Index"]:
+        if isinstance(right, numbers.Number):

Review comment:
       We should disallow `bool` literal?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847466995


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43417/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850542913


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139066/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847429108


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43410/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847293372


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43404/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r637299114



##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +41,183 @@ class BooleanOps(DataTypeOps):
     @property
     def pretty_name(self) -> str:
         return 'booleans'
+
+    def add(self, left, right):

Review comment:
       For sure, type annotations are added.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [WIP][SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845431129


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138763/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849180579


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43509/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] ueshin commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r637338960



##########
File path: python/pyspark/pandas/tests/data_type_ops/test_boolean_ops.py
##########
@@ -33,107 +35,148 @@ def pser(self):
     def psser(self):
         return ps.from_pandas(self.pser)
 
+    @property
+    def float_pser(self):
+        return pd.Series([1, 2, 3], dtype=float)
+
+    @property
+    def float_psser(self):
+        return ps.from_pandas(self.float_pser)
+
     def test_add(self):
-        self.assertRaises(TypeError, lambda: self.psser + 1)
-        self.assertRaises(TypeError, lambda: self.psser + 0.1)
+        self.assert_eq(self.pser + 1, self.psser + 1)
+        self.assert_eq(self.pser + 0.1, self.psser + 0.1)
 
         with option_context("compute.ops_on_diff_frames", True):
-            for psser in self.pssers:
+            for pser, psser in self.numeric_pser_psser_pairs:
+                self.assert_eq(self.pser + pser, (self.psser + psser).sort_index())
+
+            for psser in self.non_numeric_pssers.values():
                 self.assertRaises(TypeError, lambda: self.psser + psser)
 
     def test_sub(self):
-        self.assertRaises(TypeError, lambda: self.psser - 1)
-        self.assertRaises(TypeError, lambda: self.psser - 0.1)
+        self.assert_eq(self.pser - 1, self.psser - 1)
+        self.assert_eq(self.pser - 0.1, self.psser - 0.1)
 
         with option_context("compute.ops_on_diff_frames", True):
-            for psser in self.pssers:
+            for pser, psser in self.numeric_pser_psser_pairs:
+                self.assert_eq(self.pser - pser, (self.psser - psser).sort_index())
+
+            for psser in self.non_numeric_pssers.values():
                 self.assertRaises(TypeError, lambda: self.psser - psser)
 
     def test_mul(self):
-        self.assertRaises(TypeError, lambda: self.psser * 1)
-        self.assertRaises(TypeError, lambda: self.psser * 0.1)
+        self.assert_eq(self.pser * 1, self.psser * 1)
+        self.assert_eq(self.pser * 0.1, self.psser * 0.1)
 
         with option_context("compute.ops_on_diff_frames", True):
-            for psser in self.pssers:
+            for pser, psser in self.numeric_pser_psser_pairs:
+                self.assert_eq(self.pser * pser, (self.psser * psser).sort_index())
+
+            for psser in self.non_numeric_pssers.values():
                 self.assertRaises(TypeError, lambda: self.psser * psser)
 
     def test_truediv(self):
-        self.assertRaises(TypeError, lambda: self.psser / 1)
-        self.assertRaises(TypeError, lambda: self.psser / 0.1)
+        self.assert_eq(self.pser / 1, self.psser / 1)
+        self.assert_eq(self.pser / 0.1, self.psser / 0.1)
 
         with option_context("compute.ops_on_diff_frames", True):
-            for psser in self.pssers:
+            self.assert_eq(
+                self.pser / self.float_pser, (self.psser / self.float_psser).sort_index())
+
+            for psser in self.non_numeric_pssers.values():
                 self.assertRaises(TypeError, lambda: self.psser / psser)
 
     def test_floordiv(self):
-        self.assertRaises(TypeError, lambda: self.psser // 1)
-        self.assertRaises(TypeError, lambda: self.psser // 0.1)
+        # float is always returned in pandas-on-Spark
+        self.assert_eq((self.pser // 1).astype("float"), self.psser // 1)
+        # in pandas, 1 // 0.1 = 9.0; in pandas-on-Spark, 1 // 0.1 = 10.0
+        # self.assert_eq(self.pser // 0.1, self.psser // 0.1)
 
         with option_context("compute.ops_on_diff_frames", True):
-            for psser in self.pssers:
+            self.assert_eq(
+                self.pser // self.float_pser, (self.psser // self.float_psser).sort_index()
+            )
+
+            for psser in self.non_numeric_pssers.values():
                 self.assertRaises(TypeError, lambda: self.psser // psser)
 
     def test_mod(self):
-        self.assertRaises(TypeError, lambda: self.psser % 1)
-        self.assertRaises(TypeError, lambda: self.psser % 0.1)
+        self.assert_eq(self.pser % 1, self.psser % 1)
+        self.assert_eq(self.pser % 0.1, self.psser % 0.1)
 
         with option_context("compute.ops_on_diff_frames", True):
-            for psser in self.pssers:
+            for pser, psser in self.numeric_pser_psser_pairs:
+                self.assert_eq(self.pser % pser, (self.psser % psser).sort_index())
+
+            for psser in self.non_numeric_pssers.values():
                 self.assertRaises(TypeError, lambda: self.psser % psser)
 
     def test_pow(self):
-        self.assertRaises(TypeError, lambda: self.psser ** 1)
-        self.assertRaises(TypeError, lambda: self.psser ** 0.1)
+        # float is always returned in pandas-on-Spark
+        self.assert_eq((self.pser ** 1).astype("float"), self.psser ** 1)
+        self.assert_eq(self.pser ** 0.1, self.psser ** 0.1)
 
         with option_context("compute.ops_on_diff_frames", True):
-            for psser in self.pssers:
+            self.assert_eq(
+                self.pser ** self.float_pser, (self.psser ** self.float_psser).sort_index()
+            )
+
+            for psser in self.non_numeric_pssers.values():
                 self.assertRaises(TypeError, lambda: self.psser ** psser)
 
     def test_radd(self):
-        self.assertRaises(TypeError, lambda: 1 + self.psser)
-        self.assertRaises(TypeError, lambda: 0.1 + self.psser)
+        self.assert_eq(1 + self.pser, 1 + self.psser)
+        self.assert_eq(0.1 + self.pser, 0.1 + self.psser)
         self.assertRaises(TypeError, lambda: "x" + self.psser)
         self.assertRaises(TypeError, lambda: datetime.date(1994, 1, 1) + self.psser)
         self.assertRaises(TypeError, lambda: datetime.datetime(1994, 1, 1) + self.psser)
 
     def test_rsub(self):
-        self.assertRaises(TypeError, lambda: 1 - self.psser)
-        self.assertRaises(TypeError, lambda: 0.1 - self.psser)
+        self.assert_eq(1 - self.pser, 1 - self.psser)
+        self.assert_eq(0.1 - self.pser, 0.1 - self.psser)
         self.assertRaises(TypeError, lambda: "x" - self.psser)
         self.assertRaises(TypeError, lambda: datetime.date(1994, 1, 1) - self.psser)
         self.assertRaises(TypeError, lambda: datetime.datetime(1994, 1, 1) - self.psser)
 
     def test_rmul(self):
-        self.assertRaises(TypeError, lambda: 1 * self.psser)
-        self.assertRaises(TypeError, lambda: 0.1 * self.psser)
+        self.assert_eq(1 * self.pser, 1 * self.psser)
+        self.assert_eq(0.1 * self.pser, 0.1 * self.psser)
         self.assertRaises(TypeError, lambda: "x" * self.psser)
         self.assertRaises(TypeError, lambda: datetime.date(1994, 1, 1) * self.psser)
         self.assertRaises(TypeError, lambda: datetime.datetime(1994, 1, 1) * self.psser)
 
     def test_rtruediv(self):
-        self.assertRaises(TypeError, lambda: 1 / self.psser)
-        self.assertRaises(TypeError, lambda: 0.1 / self.psser)
+        self.assert_eq(1 / self.pser, 1 / self.psser)
+        self.assert_eq(0.1 / self.pser, 0.1 / self.psser)
         self.assertRaises(TypeError, lambda: "x" / self.psser)
         self.assertRaises(TypeError, lambda: datetime.date(1994, 1, 1) / self.psser)
         self.assertRaises(TypeError, lambda: datetime.datetime(1994, 1, 1) / self.psser)
 
     def test_rfloordiv(self):
-        self.assertRaises(TypeError, lambda: 1 // self.psser)
-        self.assertRaises(TypeError, lambda: 0.1 // self.psser)
+        if LooseVersion(pd.__version__) >= LooseVersion("0.25.3"):
+            self.assert_eq(1 // self.pser, 1 // self.psser)
+            self.assert_eq(0.1 // self.pser, 0.1 // self.psser)

Review comment:
       What happens in the `else` case?

##########
File path: python/pyspark/pandas/tests/data_type_ops/test_num_ops.py
##########
@@ -101,7 +133,13 @@ def test_floordiv(self):
                 self.assertRaises(TypeError, lambda: psser // self.non_numeric_pssers["date"])
                 self.assertRaises(
                     TypeError, lambda: psser // self.non_numeric_pssers["categorical"])
-                self.assertRaises(TypeError, lambda: psser // self.non_numeric_pssers["bool"])
+                if LooseVersion(pd.__version__) >= LooseVersion("0.25.3"):
+                    self.assert_eq(
+                        (self.float_psser // self.non_numeric_pssers["bool"]).sort_index(),
+                        self.float_pser // self.non_numeric_psers["bool"],
+                    )

Review comment:
       ditto.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r638268484



##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -50,7 +50,7 @@ def pretty_name(self) -> str:
     def add(self, left, right) -> Union["Series", "Index"]:
         if isinstance(right, numbers.Number):
             left = left.spark.transform(lambda scol: scol.cast(as_spark_type(type(right))))
-            return column_op(Column.__add__)(left, right)
+            return left + right
         elif isinstance(right, IndexOpsMixin) and is_numeric_index_ops(right):
             left = transform_boolean_operand_to_numeric(left, right.spark.data_type)
             return column_op(Column.__add__)(left, right)

Review comment:
       You are right, I was thinking it caused an infinite recursion. `left + right`s are applied.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849193160


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43509/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847209831


   **[Test build #138880 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138880/testReport)** for PR 32611 at commit [`b4e2bc7`](https://github.com/apache/spark/commit/b4e2bc7689cdac95e4358e48754f8cdb4cf313fb).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r638106867



##########
File path: python/pyspark/pandas/data_type_ops/num_ops.py
##########
@@ -16,31 +16,46 @@
 #
 
 import numbers
-from typing import TYPE_CHECKING, Union
+from typing import Any, TYPE_CHECKING, Union
 
 import numpy as np
 from pandas.api.types import CategoricalDtype
 
 from pyspark.sql import Column, functions as F
 from pyspark.sql.types import (
+    BooleanType,
     NumericType,
     StringType,
     TimestampType,
 )
 
 from pyspark.pandas.base import column_op, IndexOpsMixin, numpy_column_op
-from pyspark.pandas.data_type_ops.base import DataTypeOps
+from pyspark.pandas.data_type_ops.base import DataTypeOps, transform_boolean_operand_to_numeric
 from pyspark.pandas.spark import functions as SF
+from pyspark.sql.column import Column
+
 
 if TYPE_CHECKING:
     from pyspark.pandas.indexes import Index  # noqa: F401 (SPARK-34943)
     from pyspark.pandas.series import Series  # noqa: F401 (SPARK-34943)
 
 
+def is_valid_operand_for_numeric_arithmetic(operand: Any) -> bool:
+    """Check whether the operand is valid for arithmetic operations against numerics."""
+    if isinstance(operand, numbers.Number):
+        return True
+    elif isinstance(operand, IndexOpsMixin):
+        if isinstance(operand.dtype, CategoricalDtype):
+            return False
+        else:
+            return isinstance(operand.spark.data_type, NumericType) or isinstance(
+                operand.spark.data_type, BooleanType)
+    else:
+        return isinstance(operand, Column)

Review comment:
       Yes, we do.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846081840


   **[Test build #138809 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138809/testReport)** for PR 32611 at commit [`a123db8`](https://github.com/apache/spark/commit/a123db80c882b806c6c60fb56e83b8b5253e0b63).
    * This patch **fails Python style tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r640174272



##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +39,144 @@ class BooleanOps(DataTypeOps):
     @property
     def pretty_name(self) -> str:
         return 'booleans'
+
+    def add(self, left, right) -> Union["Series", "Index"]:
+        if not is_valid_operand_for_numeric_arithmetic(right, allow_bool=False):
+            raise TypeError(
+                "Addition can not be applied to %s and the given type." % self.pretty_name)
+
+        if isinstance(right, numbers.Number) and not isinstance(right, bool):
+            left = left.spark.transform(lambda scol: scol.cast(as_spark_type(type(right))))
+            return left + right
+        else:
+            assert isinstance(right, IndexOpsMixin)

Review comment:
       Now `spark.Column` is disallowed in `is_valid_operand_for_numeric_arithmetic`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-848083906


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43457/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847427963


   **[Test build #138891 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138891/testReport)** for PR 32611 at commit [`7ea2876`](https://github.com/apache/spark/commit/7ea2876d0c5083ffa2eddbc634edab6e4a74d71a).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846321739


   **[Test build #138818 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138818/testReport)** for PR 32611 at commit [`925e34b`](https://github.com/apache/spark/commit/925e34b3bffb6496d62425fa08d36177589cb17d).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846080849


   **[Test build #138809 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138809/testReport)** for PR 32611 at commit [`a123db8`](https://github.com/apache/spark/commit/a123db80c882b806c6c60fb56e83b8b5253e0b63).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [WIP][SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845470209


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43285/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-databricks commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845619671


   @HyukjinKwon Certainly, examples are added.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] ueshin commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r638207153



##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -50,7 +50,7 @@ def pretty_name(self) -> str:
     def add(self, left, right) -> Union["Series", "Index"]:
         if isinstance(right, numbers.Number):
             left = left.spark.transform(lambda scol: scol.cast(as_spark_type(type(right))))
-            return column_op(Column.__add__)(left, right)
+            return left + right
         elif isinstance(right, IndexOpsMixin) and is_numeric_index_ops(right):
             left = transform_boolean_operand_to_numeric(left, right.spark.data_type)
             return column_op(Column.__add__)(left, right)

Review comment:
       We can use `left + right` here as well?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] ueshin commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r637323275



##########
File path: python/pyspark/pandas/tests/data_type_ops/test_num_ops.py
##########
@@ -42,7 +54,12 @@ def test_add(self):
                 self.assertRaises(TypeError, lambda: psser + self.non_numeric_pssers["datetime"])
                 self.assertRaises(TypeError, lambda: psser + self.non_numeric_pssers["date"])
                 self.assertRaises(TypeError, lambda: psser + self.non_numeric_pssers["categorical"])
-                self.assertRaises(TypeError, lambda: psser + self.non_numeric_pssers["bool"])
+                self.assert_eq(
+                    (psser + self.non_numeric_pssers["bool"]).sort_index(),
+                    pser + self.non_numeric_psers["bool"],
+                )
+
+            self.assertRaises(TypeError, lambda: self.float_psser + True)

Review comment:
       This is supposed to be out of for-loop?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] ueshin commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r637311351



##########
File path: python/pyspark/pandas/tests/data_type_ops/test_num_ops.py
##########
@@ -30,6 +34,14 @@ class NumOpsTest(PandasOnSparkTestCase, TestCasesUtils):
     returns float32.
     The underlying reason is the respective Spark operations return DoubleType always.
     """
+    @property
+    def float_pser(self):
+        return pd.Series([1, 2, 3], dtype=float)
+
+    @property
+    def float_psser(self):
+        return ps.from_pandas(self.float_pser)
+
     def test_add(self):
         for pser, psser in self.numeric_pser_psser_pairs:
             self.assert_eq(pser + pser, psser + psser)

Review comment:
       Isn't this PR supposed to support it?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] ueshin commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r637228133



##########
File path: python/pyspark/pandas/tests/data_type_ops/test_date_ops.py
##########
@@ -60,7 +61,9 @@ def test_sub(self):
         with option_context("compute.ops_on_diff_frames", True):
             for pser, psser in self.pser_psser_pairs:
                 if isinstance(psser.spark.data_type, DateType):
-                    self.assert_eq((self.pser - pser).dt.days, (self.psser - psser).sort_index())
+                    if LooseVersion(pd.__version__) >= LooseVersion("0.24.2"):
+                        self.assert_eq(
+                            (self.pser - pser).dt.days, (self.psser - psser).sort_index())

Review comment:
       What happens in `else` case?

##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +41,183 @@ class BooleanOps(DataTypeOps):
     @property
     def pretty_name(self) -> str:
         return 'booleans'
+
+    def add(self, left, right):
+        if isinstance(right, numbers.Number):
+            left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
+            return column_op(Column.__add__)(left, right)
+        elif isinstance(right, IndexOpsMixin) and is_numeric_index_ops(right):
+            left = transform_boolean_operand_to_numeric(left, right.spark.data_type)
+            return column_op(Column.__add__)(left, right)
+        else:
+            raise TypeError(
+                "Addition can not be applied to %s and the given type." % self.pretty_name)
+
+    def sub(self, left, right):
+        if isinstance(right, numbers.Number):
+            left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
+            return column_op(Column.__sub__)(left, right)
+        elif isinstance(right, IndexOpsMixin) and is_numeric_index_ops(right):
+            left = transform_boolean_operand_to_numeric(left, right.spark.data_type)
+            return column_op(Column.__sub__)(left, right)
+        else:
+            raise TypeError(
+                "Subtraction can not be applied to %s and the given type." % self.pretty_name)
+
+    def mul(self, left, right):
+        if isinstance(right, numbers.Number):
+            left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
+            return column_op(Column.__mul__)(left, right)
+        elif isinstance(right, IndexOpsMixin) and is_numeric_index_ops(right):
+            left = transform_boolean_operand_to_numeric(left, right.spark.data_type)
+            return column_op(Column.__mul__)(left, right)
+        else:
+            raise TypeError(
+                "Multiplication can not be applied to %s and the given type." % self.pretty_name)
+
+    def truediv(self, left, right):
+        def truediv(left, right):
+            return F.when(F.lit(right != 0) | F.lit(right).isNull(), left.__div__(right)).otherwise(
+                F.when(F.lit(left == np.inf) | F.lit(left == -np.inf), left).otherwise(
+                    F.lit(np.inf).__div__(left)
+                )
+            )
+
+        if isinstance(right, numbers.Number):
+            left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
+            return numpy_column_op(truediv)(left, right)
+        elif isinstance(right, IndexOpsMixin) and is_numeric_index_ops(right):
+            left = transform_boolean_operand_to_numeric(left, right.spark.data_type)
+            return numpy_column_op(truediv)(left, right)
+        else:
+            raise TypeError(
+                "True division can not be applied to %s and the given type." % self.pretty_name)
+
+    def floordiv(self, left, right):
+        def floordiv(left, right):
+            return F.when(F.lit(right is np.nan), np.nan).otherwise(
+                F.when(
+                    F.lit(right != 0) | F.lit(right).isNull(), F.floor(left.__div__(right))
+                ).otherwise(
+                    F.when(F.lit(left == np.inf) | F.lit(left == -np.inf), left).otherwise(
+                        F.lit(np.inf).__div__(left)
+                    )
+                )
+            )
+
+        if isinstance(right, numbers.Number):
+            left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
+            return numpy_column_op(floordiv)(left, right)
+        elif isinstance(right, IndexOpsMixin) and is_numeric_index_ops(right):
+            left = transform_boolean_operand_to_numeric(left, right.spark.data_type)
+            return numpy_column_op(floordiv)(left, right)
+        else:
+            raise TypeError(
+                "Floor division can not be applied to %s and the given type." % self.pretty_name)
+
+    def mod(self, left, right):
+        def mod(left, right):
+            return ((left % right) + right) % right
+
+        if isinstance(right, numbers.Number):
+            left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
+            return column_op(mod)(left, right)
+        elif isinstance(right, IndexOpsMixin) and is_numeric_index_ops(right):
+            left = transform_boolean_operand_to_numeric(left, right.spark.data_type)
+            return column_op(mod)(left, right)
+        else:
+            raise TypeError(
+                "Modulo can not be applied to %s and the given type." % self.pretty_name)
+
+    def pow(self, left, right):
+        def pow_func(left, right):
+            return F.when(left == 1, left).otherwise(Column.__pow__(left, right))
+
+        if isinstance(right, numbers.Number):
+            left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
+            return column_op(pow_func)(left, right)
+        elif isinstance(right, IndexOpsMixin) and is_numeric_index_ops(right):
+            left = transform_boolean_operand_to_numeric(left, right.spark.data_type)
+            return column_op(pow_func)(left, right)
+
+        else:
+            raise TypeError(
+                "Exponentiation can not be applied to %s and the given type." % self.pretty_name)
+
+    def radd(self, left, right=None):
+        if isinstance(right, numbers.Number):
+            left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))
+            return column_op(Column.__radd__)(left, right)
+        else:
+            raise TypeError(
+                "Addition can not be applied to %s and the given type." % self.pretty_name)
+
+    def rsub(self, left, right=None):

Review comment:
       We don't need `=None`?

##########
File path: python/pyspark/pandas/tests/data_type_ops/test_boolean_ops.py
##########
@@ -33,107 +35,148 @@ def pser(self):
     def psser(self):
         return ps.from_pandas(self.pser)
 
+    @property
+    def float_pser(self):
+        return pd.Series([1, 2, 3], dtype=float)
+
+    @property
+    def float_psser(self):
+        return ps.from_pandas(self.float_pser)
+
     def test_add(self):
-        self.assertRaises(TypeError, lambda: self.psser + 1)
-        self.assertRaises(TypeError, lambda: self.psser + 0.1)
+        self.assert_eq(self.pser + 1, self.psser + 1)
+        self.assert_eq(self.pser + 0.1, self.psser + 0.1)

Review comment:
       What if:
   
   ```py
   psser = self.psser
   psser + psser.astype(int)
   psser + psser
   ```

##########
File path: python/pyspark/pandas/tests/data_type_ops/test_num_ops.py
##########
@@ -30,6 +34,14 @@ class NumOpsTest(PandasOnSparkTestCase, TestCasesUtils):
     returns float32.
     The underlying reason is the respective Spark operations return DoubleType always.
     """
+    @property
+    def float_pser(self):
+        return pd.Series([1, 2, 3], dtype=float)
+
+    @property
+    def float_psser(self):
+        return ps.from_pandas(self.float_pser)
+
     def test_add(self):
         for pser, psser in self.numeric_pser_psser_pairs:
             self.assert_eq(pser + pser, psser + psser)

Review comment:
       What if:
   ```py
   psser + True
   psser + psser.astype(bool)
   ```

##########
File path: python/pyspark/pandas/tests/data_type_ops/test_date_ops.py
##########
@@ -60,7 +61,9 @@ def test_sub(self):
         with option_context("compute.ops_on_diff_frames", True):
             for pser, psser in self.pser_psser_pairs:
                 if isinstance(psser.spark.data_type, DateType):
-                    self.assert_eq((self.pser - pser).dt.days, (self.psser - psser).sort_index())
+                    if LooseVersion(pd.__version__) >= LooseVersion("0.24.2"):
+                        self.assert_eq(
+                            (self.pser - pser).dt.days, (self.psser - psser).sort_index())

Review comment:
       btw, this is not related to this PR?

##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +41,183 @@ class BooleanOps(DataTypeOps):
     @property
     def pretty_name(self) -> str:
         return 'booleans'
+
+    def add(self, left, right):
+        if isinstance(right, numbers.Number):
+            left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))

Review comment:
       `left.spark.transform` instead of `apply`?

##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +41,183 @@ class BooleanOps(DataTypeOps):
     @property
     def pretty_name(self) -> str:
         return 'booleans'
+
+    def add(self, left, right):

Review comment:
       Could you add type annotations?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846226896


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43333/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846317781


   **[Test build #138818 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138818/testReport)** for PR 32611 at commit [`925e34b`](https://github.com/apache/spark/commit/925e34b3bffb6496d62425fa08d36177589cb17d).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847421527


   **[Test build #138892 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138892/testReport)** for PR 32611 at commit [`a608e5a`](https://github.com/apache/spark/commit/a608e5a57ad27236b4e412fd8e4d440937c26e61).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [WIP][SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845505274


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138766/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845618630


   **[Test build #138785 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138785/testReport)** for PR 32611 at commit [`3fef44b`](https://github.com/apache/spark/commit/3fef44bea9b90463b01eda946939faddd88740a5).
    * This patch **fails Python style tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845511899


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138768/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845535666


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138769/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845555582


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43291/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r640957039



##########
File path: python/pyspark/pandas/tests/test_dataframe.py
##########
@@ -121,7 +121,7 @@ def test_extension_dtypes(self):
         psdf = ps.from_pandas(pdf)
 
         self._check_extension(psdf, pdf)
-        self._check_extension(psdf + F.lit(1).cast("byte"), pdf + 1)
+        # self._check_extension(psdf + F.lit(1).cast("byte"), pdf + 1)

Review comment:
       @ueshin Shall we disable the test case?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849218408


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43510/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847419100






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849972655


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43551/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849933905


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139033/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r639177972



##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +39,144 @@ class BooleanOps(DataTypeOps):
     @property
     def pretty_name(self) -> str:
         return 'booleans'
+
+    def add(self, left, right) -> Union["Series", "Index"]:
+        if not is_valid_operand_for_numeric_arithmetic(right, allow_bool=False):
+            raise TypeError(
+                "Addition can not be applied to %s and the given type." % self.pretty_name)
+
+        if isinstance(right, numbers.Number) and not isinstance(right, bool):
+            left = left.spark.transform(lambda scol: scol.cast(as_spark_type(type(right))))
+            return left + right
+        else:
+            assert isinstance(right, IndexOpsMixin)

Review comment:
       I see what you mean. I don't find an existing code path for `right` to be `spark.Column` for `boolean_ops` though.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847350100


   **[Test build #138888 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138888/testReport)** for PR 32611 at commit [`99e66a1`](https://github.com/apache/spark/commit/99e66a18a07f0ba69a0c036b36374dd8be74ba49).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r638317052



##########
File path: python/pyspark/pandas/data_type_ops/num_ops.py
##########
@@ -40,16 +40,19 @@
     from pyspark.pandas.series import Series  # noqa: F401 (SPARK-34943)
 
 
-def is_valid_operand_for_numeric_arithmetic(operand: Any) -> bool:
+def is_valid_operand_for_numeric_arithmetic(
+    operand: Any,
+    allow_bool_index_ops: bool = True
+) -> bool:

Review comment:
       Good idea, `allow_bool` is set as a keyword-only argument.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r643361468



##########
File path: python/pyspark/pandas/tests/data_type_ops/test_num_ops.py
##########
@@ -30,6 +34,14 @@ class NumOpsTest(PandasOnSparkTestCase, TestCasesUtils):
     returns float32.
     The underlying reason is the respective Spark operations return DoubleType always.
     """
+    @property
+    def float_pser(self):
+        return pd.Series([1, 2, 3], dtype=float)
+
+    @property
+    def float_psser(self):
+        return ps.from_pandas(self.float_pser)
+
     def test_add(self):
         for pser, psser in self.numeric_pser_psser_pairs:
             self.assert_eq(pser + pser, psser + psser)

Review comment:
       https://issues.apache.org/jira/browse/SPARK-35601 is created for that.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847390918


   **[Test build #138889 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138889/testReport)** for PR 32611 at commit [`7dac0d1`](https://github.com/apache/spark/commit/7dac0d1502f9265b118ed54dae4bdd69aadd6d32).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850542195


   **[Test build #139066 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139066/testReport)** for PR 32611 at commit [`7b6d6a6`](https://github.com/apache/spark/commit/7b6d6a6986d9401692911ef7512ab260328dc115).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850707017


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139073/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849164895


   **[Test build #138990 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138990/testReport)** for PR 32611 at commit [`98a5c8f`](https://github.com/apache/spark/commit/98a5c8ff8bec0dcfcb386fd970fedf55384f1571).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845618215


   **[Test build #138785 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138785/testReport)** for PR 32611 at commit [`3fef44b`](https://github.com/apache/spark/commit/3fef44bea9b90463b01eda946939faddd88740a5).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845702336


   Kubernetes integration test unable to build dist.
   
   exiting with code: 1
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43311/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847346303


   **[Test build #138885 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138885/testReport)** for PR 32611 at commit [`5e2f911`](https://github.com/apache/spark/commit/5e2f91152dc0ad45c7155d94e4b87e82523ed04b).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847466985


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43417/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847403018


   **[Test build #138889 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138889/testReport)** for PR 32611 at commit [`7dac0d1`](https://github.com/apache/spark/commit/7dac0d1502f9265b118ed54dae4bdd69aadd6d32).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847270458


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43402/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-848024627


   **[Test build #138936 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138936/testReport)** for PR 32611 at commit [`9a8f0b9`](https://github.com/apache/spark/commit/9a8f0b9a693281b5c66509c40e97a9525a216835).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846328527


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43340/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847390479


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138888/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849164895


   **[Test build #138990 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138990/testReport)** for PR 32611 at commit [`98a5c8f`](https://github.com/apache/spark/commit/98a5c8ff8bec0dcfcb386fd970fedf55384f1571).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847242145


   **[Test build #138882 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138882/testReport)** for PR 32611 at commit [`43a2972`](https://github.com/apache/spark/commit/43a29721e982df716a874286347e8b93492e1fd7).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845509991


   **[Test build #138768 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138768/testReport)** for PR 32611 at commit [`6c5a5cf`](https://github.com/apache/spark/commit/6c5a5cf431e0e23b67fdcbee578563dffb366947).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r637298978



##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +41,183 @@ class BooleanOps(DataTypeOps):
     @property
     def pretty_name(self) -> str:
         return 'booleans'
+
+    def add(self, left, right):
+        if isinstance(right, numbers.Number):
+            left = left.spark.apply(lambda scol: scol.cast(as_spark_type(type(right))))

Review comment:
       Cool! Updated.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845652569


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43308/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-848070211


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138936/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r640174525



##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +39,144 @@ class BooleanOps(DataTypeOps):
     @property
     def pretty_name(self) -> str:
         return 'booleans'
+
+    def add(self, left, right) -> Union["Series", "Index"]:
+        if not is_valid_operand_for_numeric_arithmetic(right, allow_bool=False):
+            raise TypeError(
+                "Addition can not be applied to %s and the given type." % self.pretty_name)
+
+        if isinstance(right, numbers.Number) and not isinstance(right, bool):
+            left = left.spark.transform(lambda scol: scol.cast(as_spark_type(type(right))))
+            return left + right
+        else:
+            assert isinstance(right, IndexOpsMixin)

Review comment:
       Shall we disable `spark.Column` as operands for other Ops class in a separate PR?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [WIP][SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845470209


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43285/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845511899


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138768/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846111969


   **[Test build #138811 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138811/testReport)** for PR 32611 at commit [`bd63058`](https://github.com/apache/spark/commit/bd630589cc6d686315053e6024cc61f4fb396bfd).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] ueshin commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r638314268



##########
File path: python/pyspark/pandas/data_type_ops/num_ops.py
##########
@@ -40,16 +40,19 @@
     from pyspark.pandas.series import Series  # noqa: F401 (SPARK-34943)
 
 
-def is_valid_operand_for_numeric_arithmetic(operand: Any) -> bool:
+def is_valid_operand_for_numeric_arithmetic(
+    operand: Any,
+    allow_bool_index_ops: bool = True
+) -> bool:

Review comment:
       Btw, the argument name should be like `allow_bool` or something since we will want to use it for bool literal as an operand in a following PR.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] ueshin commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r638212053



##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -15,7 +15,27 @@
 # limitations under the License.
 #
 
-from pyspark.pandas.data_type_ops.base import DataTypeOps
+import numbers
+from typing import TYPE_CHECKING, Union
+
+import numpy as np
+from pandas.api.types import CategoricalDtype
+
+from pyspark.pandas.base import column_op, IndexOpsMixin, numpy_column_op
+from pyspark.pandas.data_type_ops.base import DataTypeOps, transform_boolean_operand_to_numeric
+from pyspark.pandas.typedef.typehints import as_spark_type
+from pyspark.sql import Column, functions as F
+from pyspark.sql.types import NumericType
+
+if TYPE_CHECKING:
+    from pyspark.pandas.indexes import Index  # noqa: F401 (SPARK-34943)
+    from pyspark.pandas.series import Series  # noqa: F401 (SPARK-34943)
+
+
+def is_numeric_index_ops(index_ops: IndexOpsMixin) -> bool:
+    """Check if the given index_ops is numeric IndexOpsMixin."""
+    return isinstance(index_ops.spark.data_type, NumericType) and (
+        not isinstance(index_ops.dtype, CategoricalDtype))

Review comment:
       Sounds good.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846080849


   **[Test build #138809 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138809/testReport)** for PR 32611 at commit [`a123db8`](https://github.com/apache/spark/commit/a123db80c882b806c6c60fb56e83b8b5253e0b63).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845707439


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43311/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845557970


   Kubernetes integration test unable to build dist.
   
   exiting with code: 1
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43292/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849193160


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43509/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849188150


   **[Test build #138991 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138991/testReport)** for PR 32611 at commit [`5905012`](https://github.com/apache/spark/commit/5905012338a16ba500094b9cae33bb783e86faff).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847310471


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43404/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-848110079


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43457/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845540848


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43291/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847417170


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43411/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847442495


   **[Test build #138895 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138895/testReport)** for PR 32611 at commit [`203f90a`](https://github.com/apache/spark/commit/203f90aab81cecf2591f2fe2aa4b5466b2e486ae).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849211282


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138991/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r638166139



##########
File path: python/pyspark/pandas/tests/data_type_ops/test_boolean_ops.py
##########
@@ -33,107 +35,148 @@ def pser(self):
     def psser(self):
         return ps.from_pandas(self.pser)
 
+    @property
+    def float_pser(self):
+        return pd.Series([1, 2, 3], dtype=float)
+
+    @property
+    def float_psser(self):
+        return ps.from_pandas(self.float_pser)
+
     def test_add(self):
-        self.assertRaises(TypeError, lambda: self.psser + 1)
-        self.assertRaises(TypeError, lambda: self.psser + 0.1)
+        self.assert_eq(self.pser + 1, self.psser + 1)
+        self.assert_eq(self.pser + 0.1, self.psser + 0.1)
 
         with option_context("compute.ops_on_diff_frames", True):
-            for psser in self.pssers:
+            for pser, psser in self.numeric_pser_psser_pairs:
+                self.assert_eq(self.pser + pser, (self.psser + psser).sort_index())
+
+            for psser in self.non_numeric_pssers.values():
                 self.assertRaises(TypeError, lambda: self.psser + psser)
 
     def test_sub(self):
-        self.assertRaises(TypeError, lambda: self.psser - 1)
-        self.assertRaises(TypeError, lambda: self.psser - 0.1)
+        self.assert_eq(self.pser - 1, self.psser - 1)
+        self.assert_eq(self.pser - 0.1, self.psser - 0.1)
 
         with option_context("compute.ops_on_diff_frames", True):
-            for psser in self.pssers:
+            for pser, psser in self.numeric_pser_psser_pairs:
+                self.assert_eq(self.pser - pser, (self.psser - psser).sort_index())
+
+            for psser in self.non_numeric_pssers.values():
                 self.assertRaises(TypeError, lambda: self.psser - psser)
 
     def test_mul(self):
-        self.assertRaises(TypeError, lambda: self.psser * 1)
-        self.assertRaises(TypeError, lambda: self.psser * 0.1)
+        self.assert_eq(self.pser * 1, self.psser * 1)
+        self.assert_eq(self.pser * 0.1, self.psser * 0.1)
 
         with option_context("compute.ops_on_diff_frames", True):
-            for psser in self.pssers:
+            for pser, psser in self.numeric_pser_psser_pairs:
+                self.assert_eq(self.pser * pser, (self.psser * psser).sort_index())
+
+            for psser in self.non_numeric_pssers.values():
                 self.assertRaises(TypeError, lambda: self.psser * psser)
 
     def test_truediv(self):
-        self.assertRaises(TypeError, lambda: self.psser / 1)
-        self.assertRaises(TypeError, lambda: self.psser / 0.1)
+        self.assert_eq(self.pser / 1, self.psser / 1)
+        self.assert_eq(self.pser / 0.1, self.psser / 0.1)
 
         with option_context("compute.ops_on_diff_frames", True):
-            for psser in self.pssers:
+            self.assert_eq(
+                self.pser / self.float_pser, (self.psser / self.float_psser).sort_index())
+
+            for psser in self.non_numeric_pssers.values():
                 self.assertRaises(TypeError, lambda: self.psser / psser)
 
     def test_floordiv(self):
-        self.assertRaises(TypeError, lambda: self.psser // 1)
-        self.assertRaises(TypeError, lambda: self.psser // 0.1)
+        # float is always returned in pandas-on-Spark
+        self.assert_eq((self.pser // 1).astype("float"), self.psser // 1)
+        # in pandas, 1 // 0.1 = 9.0; in pandas-on-Spark, 1 // 0.1 = 10.0
+        # self.assert_eq(self.pser // 0.1, self.psser // 0.1)
 
         with option_context("compute.ops_on_diff_frames", True):
-            for psser in self.pssers:
+            self.assert_eq(
+                self.pser // self.float_pser, (self.psser // self.float_psser).sort_index()
+            )
+
+            for psser in self.non_numeric_pssers.values():
                 self.assertRaises(TypeError, lambda: self.psser // psser)
 
     def test_mod(self):
-        self.assertRaises(TypeError, lambda: self.psser % 1)
-        self.assertRaises(TypeError, lambda: self.psser % 0.1)
+        self.assert_eq(self.pser % 1, self.psser % 1)
+        self.assert_eq(self.pser % 0.1, self.psser % 0.1)
 
         with option_context("compute.ops_on_diff_frames", True):
-            for psser in self.pssers:
+            for pser, psser in self.numeric_pser_psser_pairs:
+                self.assert_eq(self.pser % pser, (self.psser % psser).sort_index())
+
+            for psser in self.non_numeric_pssers.values():
                 self.assertRaises(TypeError, lambda: self.psser % psser)
 
     def test_pow(self):
-        self.assertRaises(TypeError, lambda: self.psser ** 1)
-        self.assertRaises(TypeError, lambda: self.psser ** 0.1)
+        # float is always returned in pandas-on-Spark
+        self.assert_eq((self.pser ** 1).astype("float"), self.psser ** 1)
+        self.assert_eq(self.pser ** 0.1, self.psser ** 0.1)
 
         with option_context("compute.ops_on_diff_frames", True):
-            for psser in self.pssers:
+            self.assert_eq(
+                self.pser ** self.float_pser, (self.psser ** self.float_psser).sort_index()
+            )
+
+            for psser in self.non_numeric_pssers.values():
                 self.assertRaises(TypeError, lambda: self.psser ** psser)
 
     def test_radd(self):
-        self.assertRaises(TypeError, lambda: 1 + self.psser)
-        self.assertRaises(TypeError, lambda: 0.1 + self.psser)
+        self.assert_eq(1 + self.pser, 1 + self.psser)
+        self.assert_eq(0.1 + self.pser, 0.1 + self.psser)
         self.assertRaises(TypeError, lambda: "x" + self.psser)
         self.assertRaises(TypeError, lambda: datetime.date(1994, 1, 1) + self.psser)
         self.assertRaises(TypeError, lambda: datetime.datetime(1994, 1, 1) + self.psser)
 
     def test_rsub(self):
-        self.assertRaises(TypeError, lambda: 1 - self.psser)
-        self.assertRaises(TypeError, lambda: 0.1 - self.psser)
+        self.assert_eq(1 - self.pser, 1 - self.psser)
+        self.assert_eq(0.1 - self.pser, 0.1 - self.psser)
         self.assertRaises(TypeError, lambda: "x" - self.psser)
         self.assertRaises(TypeError, lambda: datetime.date(1994, 1, 1) - self.psser)
         self.assertRaises(TypeError, lambda: datetime.datetime(1994, 1, 1) - self.psser)
 
     def test_rmul(self):
-        self.assertRaises(TypeError, lambda: 1 * self.psser)
-        self.assertRaises(TypeError, lambda: 0.1 * self.psser)
+        self.assert_eq(1 * self.pser, 1 * self.psser)
+        self.assert_eq(0.1 * self.pser, 0.1 * self.psser)
         self.assertRaises(TypeError, lambda: "x" * self.psser)
         self.assertRaises(TypeError, lambda: datetime.date(1994, 1, 1) * self.psser)
         self.assertRaises(TypeError, lambda: datetime.datetime(1994, 1, 1) * self.psser)
 
     def test_rtruediv(self):
-        self.assertRaises(TypeError, lambda: 1 / self.psser)
-        self.assertRaises(TypeError, lambda: 0.1 / self.psser)
+        self.assert_eq(1 / self.pser, 1 / self.psser)
+        self.assert_eq(0.1 / self.pser, 0.1 / self.psser)
         self.assertRaises(TypeError, lambda: "x" / self.psser)
         self.assertRaises(TypeError, lambda: datetime.date(1994, 1, 1) / self.psser)
         self.assertRaises(TypeError, lambda: datetime.datetime(1994, 1, 1) / self.psser)
 
     def test_rfloordiv(self):
-        self.assertRaises(TypeError, lambda: 1 // self.psser)
-        self.assertRaises(TypeError, lambda: 0.1 // self.psser)
+        if LooseVersion(pd.__version__) >= LooseVersion("0.25.3"):
+            self.assert_eq(1 // self.pser, 1 // self.psser)
+            self.assert_eq(0.1 // self.pser, 0.1 // self.psser)

Review comment:
       `else` case is added.

##########
File path: python/pyspark/pandas/tests/data_type_ops/test_num_ops.py
##########
@@ -101,7 +133,13 @@ def test_floordiv(self):
                 self.assertRaises(TypeError, lambda: psser // self.non_numeric_pssers["date"])
                 self.assertRaises(
                     TypeError, lambda: psser // self.non_numeric_pssers["categorical"])
-                self.assertRaises(TypeError, lambda: psser // self.non_numeric_pssers["bool"])
+                if LooseVersion(pd.__version__) >= LooseVersion("0.25.3"):
+                    self.assert_eq(
+                        (self.float_psser // self.non_numeric_pssers["bool"]).sort_index(),
+                        self.float_pser // self.non_numeric_psers["bool"],
+                    )

Review comment:
       `else` case is added.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r638336163



##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +38,136 @@ class BooleanOps(DataTypeOps):
     @property
     def pretty_name(self) -> str:
         return 'booleans'
+
+    def add(self, left, right) -> Union["Series", "Index"]:
+        if isinstance(right, numbers.Number):

Review comment:
       Makes sense, disabled.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r638131359



##########
File path: python/pyspark/pandas/tests/data_type_ops/test_num_ops.py
##########
@@ -30,6 +34,14 @@ class NumOpsTest(PandasOnSparkTestCase, TestCasesUtils):
     returns float32.
     The underlying reason is the respective Spark operations return DoubleType always.
     """
+    @property
+    def float_pser(self):
+        return pd.Series([1, 2, 3], dtype=float)
+
+    @property
+    def float_psser(self):
+        return ps.from_pandas(self.float_pser)
+
     def test_add(self):
         for pser, psser in self.numeric_pser_psser_pairs:
             self.assert_eq(pser + pser, psser + psser)

Review comment:
       Arithmetic operations between `psser` and `psser.astype(bool)` are tested.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r637301587



##########
File path: python/pyspark/pandas/tests/data_type_ops/test_date_ops.py
##########
@@ -60,7 +61,9 @@ def test_sub(self):
         with option_context("compute.ops_on_diff_frames", True):
             for pser, psser in self.pser_psser_pairs:
                 if isinstance(psser.spark.data_type, DateType):
-                    self.assert_eq((self.pser - pser).dt.days, (self.psser - psser).sort_index())
+                    if LooseVersion(pd.__version__) >= LooseVersion("0.24.2"):
+                        self.assert_eq(
+                            (self.pser - pser).dt.days, (self.psser - psser).sort_index())

Review comment:
       It raises weird TypeError for pandas in the `else` case, as https://github.com/databricks/koalas/runs/2568962950.
   
   No, the change is not related to this PR. Let me remove it for now.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846144301






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845652569


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43308/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] ueshin closed pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
ueshin closed pull request #32611:
URL: https://github.com/apache/spark/pull/32611


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845556717






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r638132565



##########
File path: python/pyspark/pandas/tests/data_type_ops/test_num_ops.py
##########
@@ -42,7 +54,12 @@ def test_add(self):
                 self.assertRaises(TypeError, lambda: psser + self.non_numeric_pssers["datetime"])
                 self.assertRaises(TypeError, lambda: psser + self.non_numeric_pssers["date"])
                 self.assertRaises(TypeError, lambda: psser + self.non_numeric_pssers["categorical"])
-                self.assertRaises(TypeError, lambda: psser + self.non_numeric_pssers["bool"])
+                self.assert_eq(
+                    (psser + self.non_numeric_pssers["bool"]).sort_index(),
+                    pser + self.non_numeric_psers["bool"],
+                )
+
+            self.assertRaises(TypeError, lambda: self.float_psser + True)

Review comment:
       Goot catch! It is out of `with option_context` now.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846142367


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43333/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847350100


   **[Test build #138888 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138888/testReport)** for PR 32611 at commit [`99e66a1`](https://github.com/apache/spark/commit/99e66a18a07f0ba69a0c036b36374dd8be74ba49).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847428962


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43411/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847346980


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138885/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849187308


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138990/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847230904


   **[Test build #138880 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138880/testReport)** for PR 32611 at commit [`b4e2bc7`](https://github.com/apache/spark/commit/b4e2bc7689cdac95e4358e48754f8cdb4cf313fb).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847439897


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43414/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849218408


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43510/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] ueshin commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r639195854



##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +39,144 @@ class BooleanOps(DataTypeOps):
     @property
     def pretty_name(self) -> str:
         return 'booleans'
+
+    def add(self, left, right) -> Union["Series", "Index"]:
+        if not is_valid_operand_for_numeric_arithmetic(right, allow_bool=False):
+            raise TypeError(
+                "Addition can not be applied to %s and the given type." % self.pretty_name)
+
+        if isinstance(right, numbers.Number) and not isinstance(right, bool):
+            left = left.spark.transform(lambda scol: scol.cast(as_spark_type(type(right))))
+            return left + right
+        else:
+            assert isinstance(right, IndexOpsMixin)

Review comment:
       You mentioned that `spark.Column` is allowed to be operands (https://github.com/apache/spark/pull/32611#discussion_r636607352), and if so, we should handle it properly.
   
   > And mypy likes this. Would you prefer to use cast instead?
   What do you mean?

##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +39,144 @@ class BooleanOps(DataTypeOps):
     @property
     def pretty_name(self) -> str:
         return 'booleans'
+
+    def add(self, left, right) -> Union["Series", "Index"]:
+        if not is_valid_operand_for_numeric_arithmetic(right, allow_bool=False):
+            raise TypeError(
+                "Addition can not be applied to %s and the given type." % self.pretty_name)
+
+        if isinstance(right, numbers.Number) and not isinstance(right, bool):
+            left = left.spark.transform(lambda scol: scol.cast(as_spark_type(type(right))))
+            return left + right
+        else:
+            assert isinstance(right, IndexOpsMixin)

Review comment:
       You mentioned that `spark.Column` is allowed to be operands (https://github.com/apache/spark/pull/32611#discussion_r636607352), and if so, we should handle it properly.
   
   > And mypy likes this. Would you prefer to use cast instead?
   
   What do you mean?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850711760


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43594/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [WIP][SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845431129


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138763/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850542195


   **[Test build #139066 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139066/testReport)** for PR 32611 at commit [`7b6d6a6`](https://github.com/apache/spark/commit/7b6d6a6986d9401692911ef7512ab260328dc115).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847466995


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43417/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] ueshin commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r637338469



##########
File path: python/pyspark/pandas/data_type_ops/num_ops.py
##########
@@ -16,31 +16,46 @@
 #
 
 import numbers
-from typing import TYPE_CHECKING, Union
+from typing import Any, TYPE_CHECKING, Union
 
 import numpy as np
 from pandas.api.types import CategoricalDtype
 
 from pyspark.sql import Column, functions as F
 from pyspark.sql.types import (
+    BooleanType,
     NumericType,
     StringType,
     TimestampType,
 )
 
 from pyspark.pandas.base import column_op, IndexOpsMixin, numpy_column_op
-from pyspark.pandas.data_type_ops.base import DataTypeOps
+from pyspark.pandas.data_type_ops.base import DataTypeOps, transform_boolean_operand_to_numeric
 from pyspark.pandas.spark import functions as SF
+from pyspark.sql.column import Column
+
 
 if TYPE_CHECKING:
     from pyspark.pandas.indexes import Index  # noqa: F401 (SPARK-34943)
     from pyspark.pandas.series import Series  # noqa: F401 (SPARK-34943)
 
 
+def is_valid_operand_for_numeric_arithmetic(operand: Any) -> bool:
+    """Check whether the operand is valid for arithmetic operations against numerics."""
+    if isinstance(operand, numbers.Number):
+        return True
+    elif isinstance(operand, IndexOpsMixin):
+        if isinstance(operand.dtype, CategoricalDtype):
+            return False
+        else:
+            return isinstance(operand.spark.data_type, NumericType) or isinstance(
+                operand.spark.data_type, BooleanType)
+    else:
+        return isinstance(operand, Column)

Review comment:
       Do we currently allow it?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r641682948



##########
File path: python/pyspark/pandas/tests/test_dataframe.py
##########
@@ -121,7 +121,7 @@ def test_extension_dtypes(self):
         psdf = ps.from_pandas(pdf)
 
         self._check_extension(psdf, pdf)
-        self._check_extension(psdf + F.lit(1).cast("byte"), pdf + 1)
+        # self._check_extension(psdf + F.lit(1).cast("byte"), pdf + 1)

Review comment:
       Cool.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845577570


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43289/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #32611: [WIP][SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845389964


   **[Test build #138763 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138763/testReport)** for PR 32611 at commit [`d44a3b4`](https://github.com/apache/spark/commit/d44a3b4512613f071c2bd48d3aa8cd06be78a07d).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845615298


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43288/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846226896


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43333/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [WIP][SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845448415


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43285/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846144303






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847431065


   **[Test build #138892 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138892/testReport)** for PR 32611 at commit [`a608e5a`](https://github.com/apache/spark/commit/a608e5a57ad27236b4e412fd8e4d440937c26e61).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846111969


   **[Test build #138811 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138811/testReport)** for PR 32611 at commit [`bd63058`](https://github.com/apache/spark/commit/bd630589cc6d686315053e6024cc61f4fb396bfd).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846326242


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138818/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847390479


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138888/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847441076






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-848048563


   **[Test build #138936 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138936/testReport)** for PR 32611 at commit [`9a8f0b9`](https://github.com/apache/spark/commit/9a8f0b9a693281b5c66509c40e97a9525a216835).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845535666


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138769/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847242145


   **[Test build #138882 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138882/testReport)** for PR 32611 at commit [`43a2972`](https://github.com/apache/spark/commit/43a29721e982df716a874286347e8b93492e1fd7).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847209831


   **[Test build #138880 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138880/testReport)** for PR 32611 at commit [`b4e2bc7`](https://github.com/apache/spark/commit/b4e2bc7689cdac95e4358e48754f8cdb4cf313fb).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] ueshin commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r637328527



##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +46,183 @@ class BooleanOps(DataTypeOps):
     @property
     def pretty_name(self) -> str:
         return 'booleans'
+
+    def add(self, left, right) -> Union["Series", "Index"]:
+        if isinstance(right, numbers.Number):
+            left = left.spark.transform(lambda scol: scol.cast(as_spark_type(type(right))))
+            return column_op(Column.__add__)(left, right)

Review comment:
       I guess `return left + right` should be fine. The `DataTypeOps` related to `left` after converted should know how to handle it better.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846324042


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43340/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850566522


   Kubernetes integration test unable to build dist.
   
   exiting with code: 1
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43587/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r638140558



##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -15,7 +15,27 @@
 # limitations under the License.
 #
 
-from pyspark.pandas.data_type_ops.base import DataTypeOps
+import numbers
+from typing import TYPE_CHECKING, Union
+
+import numpy as np
+from pandas.api.types import CategoricalDtype
+
+from pyspark.pandas.base import column_op, IndexOpsMixin, numpy_column_op
+from pyspark.pandas.data_type_ops.base import DataTypeOps, transform_boolean_operand_to_numeric
+from pyspark.pandas.typedef.typehints import as_spark_type
+from pyspark.sql import Column, functions as F
+from pyspark.sql.types import NumericType
+
+if TYPE_CHECKING:
+    from pyspark.pandas.indexes import Index  # noqa: F401 (SPARK-34943)
+    from pyspark.pandas.series import Series  # noqa: F401 (SPARK-34943)
+
+
+def is_numeric_index_ops(index_ops: IndexOpsMixin) -> bool:
+    """Check if the given index_ops is numeric IndexOpsMixin."""
+    return isinstance(index_ops.spark.data_type, NumericType) and (
+        not isinstance(index_ops.dtype, CategoricalDtype))

Review comment:
       `is_valid_operand_for_numeric_arithmetic`  returns True for `BooleanType`, whereas `is_numeric_index_ops ` returns False for `BooleanType`.
   
   We may also pass in a boolean flag deciding how to deal with `BooleanType`. How do you think about this?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845618643


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138785/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] ueshin commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r637311351



##########
File path: python/pyspark/pandas/tests/data_type_ops/test_num_ops.py
##########
@@ -30,6 +34,14 @@ class NumOpsTest(PandasOnSparkTestCase, TestCasesUtils):
     returns float32.
     The underlying reason is the respective Spark operations return DoubleType always.
     """
+    @property
+    def float_pser(self):
+        return pd.Series([1, 2, 3], dtype=float)
+
+    @property
+    def float_psser(self):
+        return ps.from_pandas(self.float_pser)
+
     def test_add(self):
         for pser, psser in self.numeric_pser_psser_pairs:
             self.assert_eq(pser + pser, psser + psser)

Review comment:
       Isn't this PR supposed to support it?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-848070211


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138936/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845618215


   **[Test build #138785 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138785/testReport)** for PR 32611 at commit [`3fef44b`](https://github.com/apache/spark/commit/3fef44bea9b90463b01eda946939faddd88740a5).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-848110079


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43457/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847277529






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846326242


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138818/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849174414


   **[Test build #138990 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138990/testReport)** for PR 32611 at commit [`98a5c8f`](https://github.com/apache/spark/commit/98a5c8ff8bec0dcfcb386fd970fedf55384f1571).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847417614






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [WIP][SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845389964


   **[Test build #138763 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138763/testReport)** for PR 32611 at commit [`d44a3b4`](https://github.com/apache/spark/commit/d44a3b4512613f071c2bd48d3aa8cd06be78a07d).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r639177972



##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +39,144 @@ class BooleanOps(DataTypeOps):
     @property
     def pretty_name(self) -> str:
         return 'booleans'
+
+    def add(self, left, right) -> Union["Series", "Index"]:
+        if not is_valid_operand_for_numeric_arithmetic(right, allow_bool=False):
+            raise TypeError(
+                "Addition can not be applied to %s and the given type." % self.pretty_name)
+
+        if isinstance(right, numbers.Number) and not isinstance(right, bool):
+            left = left.spark.transform(lambda scol: scol.cast(as_spark_type(type(right))))
+            return left + right
+        else:
+            assert isinstance(right, IndexOpsMixin)

Review comment:
       I see what you mean. I don't find an existing code path for `right` to be `spark.Column` for `boolean_ops` though. And `mypy` likes this. Would you prefer to use `cast` instead?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847384479


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43407/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847461342


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43414/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849933881


   **[Test build #139033 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139033/testReport)** for PR 32611 at commit [`adfc5cc`](https://github.com/apache/spark/commit/adfc5cc6654a51105ef834525cdf1ebcdcd0a7bb).
    * This patch **fails Python style tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850542883


   **[Test build #139066 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139066/testReport)** for PR 32611 at commit [`7b6d6a6`](https://github.com/apache/spark/commit/7b6d6a6986d9401692911ef7512ab260328dc115).
    * This patch **fails Python style tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845652182


   Kubernetes integration test unable to build dist.
   
   exiting with code: 1
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43308/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845543774


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43289/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845635027


   **[Test build #138788 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138788/testReport)** for PR 32611 at commit [`e626c52`](https://github.com/apache/spark/commit/e626c52245521fdeb0ee46c68950605f87a987bd).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847277529






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847461342


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43414/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850703378


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43594/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847451920


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43414/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845535649


   **[Test build #138769 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138769/testReport)** for PR 32611 at commit [`b0b2a3b`](https://github.com/apache/spark/commit/b0b2a3bb61680720bb2c1be86c87a2f8dfa847d9).
    * This patch **fails Python style tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #32611: [WIP][SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845504281


   **[Test build #138766 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138766/testReport)** for PR 32611 at commit [`3f0fc27`](https://github.com/apache/spark/commit/3f0fc27ffe093ff6ef12c3f5214f6fa98c635860).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [WIP][SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845505274


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138766/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] ueshin commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r638307070



##########
File path: python/pyspark/pandas/data_type_ops/num_ops.py
##########
@@ -40,16 +40,19 @@
     from pyspark.pandas.series import Series  # noqa: F401 (SPARK-34943)
 
 
-def is_valid_operand_for_numeric_arithmetic(operand: Any) -> bool:
+def is_valid_operand_for_numeric_arithmetic(
+    operand: Any,
+    allow_bool_index_ops: bool = True
+) -> bool:

Review comment:
       Shall we make `allow_bool_index_ops` a keyword-only argument?
   
   ```py
   def is_valid_operand_for_numeric_arithmetic(
       operand: Any,
       *,
       allow_bool_index_ops: bool = True
   ) -> bool:
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847310471


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43404/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850699633


   **[Test build #139073 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139073/testReport)** for PR 32611 at commit [`db12fb1`](https://github.com/apache/spark/commit/db12fb1c854051b48caf6a253ef6dcb70694db51).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849933240


   **[Test build #139033 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139033/testReport)** for PR 32611 at commit [`adfc5cc`](https://github.com/apache/spark/commit/adfc5cc6654a51105ef834525cdf1ebcdcd0a7bb).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847275778


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43404/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845575884


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43289/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845615298


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43288/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] ueshin commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r637332388



##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -15,7 +15,27 @@
 # limitations under the License.
 #
 
-from pyspark.pandas.data_type_ops.base import DataTypeOps
+import numbers
+from typing import TYPE_CHECKING, Union
+
+import numpy as np
+from pandas.api.types import CategoricalDtype
+
+from pyspark.pandas.base import column_op, IndexOpsMixin, numpy_column_op
+from pyspark.pandas.data_type_ops.base import DataTypeOps, transform_boolean_operand_to_numeric
+from pyspark.pandas.typedef.typehints import as_spark_type
+from pyspark.sql import Column, functions as F
+from pyspark.sql.types import NumericType
+
+if TYPE_CHECKING:
+    from pyspark.pandas.indexes import Index  # noqa: F401 (SPARK-34943)
+    from pyspark.pandas.series import Series  # noqa: F401 (SPARK-34943)
+
+
+def is_numeric_index_ops(index_ops: IndexOpsMixin) -> bool:
+    """Check if the given index_ops is numeric IndexOpsMixin."""
+    return isinstance(index_ops.spark.data_type, NumericType) and (
+        not isinstance(index_ops.dtype, CategoricalDtype))

Review comment:
       I guess we can reuse `is_valid_operand_for_numeric_arithmetic` in `num_ops.py`?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849218398


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43510/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849211282


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138991/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845509991


   **[Test build #138768 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138768/testReport)** for PR 32611 at commit [`6c5a5cf`](https://github.com/apache/spark/commit/6c5a5cf431e0e23b67fdcbee578563dffb366947).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850690763


   **[Test build #139073 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139073/testReport)** for PR 32611 at commit [`db12fb1`](https://github.com/apache/spark/commit/db12fb1c854051b48caf6a253ef6dcb70694db51).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849193141


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43509/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846328532


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43340/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-848024627


   **[Test build #138936 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138936/testReport)** for PR 32611 at commit [`9a8f0b9`](https://github.com/apache/spark/commit/9a8f0b9a693281b5c66509c40e97a9525a216835).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847397923


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43407/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [WIP][SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845419493


   **[Test build #138763 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138763/testReport)** for PR 32611 at commit [`d44a3b4`](https://github.com/apache/spark/commit/d44a3b4512613f071c2bd48d3aa8cd06be78a07d).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] ueshin commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
ueshin commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-852331508


   Thanks! merging to master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846132387


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43332/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847346303


   **[Test build #138885 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138885/testReport)** for PR 32611 at commit [`5e2f911`](https://github.com/apache/spark/commit/5e2f91152dc0ad45c7155d94e4b87e82523ed04b).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846081872


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138809/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845585115


   @xinrong-databricks can we show some examples in the PR description?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849933240


   **[Test build #139033 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139033/testReport)** for PR 32611 at commit [`adfc5cc`](https://github.com/apache/spark/commit/adfc5cc6654a51105ef834525cdf1ebcdcd0a7bb).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850725471


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43594/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847390918


   **[Test build #138889 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138889/testReport)** for PR 32611 at commit [`7dac0d1`](https://github.com/apache/spark/commit/7dac0d1502f9265b118ed54dae4bdd69aadd6d32).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-databricks commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846081226


   CC @ueshin @HyukjinKwon @itholic 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [WIP][SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845504281


   **[Test build #138766 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138766/testReport)** for PR 32611 at commit [`3f0fc27`](https://github.com/apache/spark/commit/3f0fc27ffe093ff6ef12c3f5214f6fa98c635860).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846113323


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43332/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847442495


   **[Test build #138895 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138895/testReport)** for PR 32611 at commit [`203f90a`](https://github.com/apache/spark/commit/203f90aab81cecf2591f2fe2aa4b5466b2e486ae).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r638127919



##########
File path: python/pyspark/pandas/tests/data_type_ops/test_boolean_ops.py
##########
@@ -33,107 +35,148 @@ def pser(self):
     def psser(self):
         return ps.from_pandas(self.pser)
 
+    @property
+    def float_pser(self):
+        return pd.Series([1, 2, 3], dtype=float)
+
+    @property
+    def float_psser(self):
+        return ps.from_pandas(self.float_pser)
+
     def test_add(self):
-        self.assertRaises(TypeError, lambda: self.psser + 1)
-        self.assertRaises(TypeError, lambda: self.psser + 0.1)
+        self.assert_eq(self.pser + 1, self.psser + 1)
+        self.assert_eq(self.pser + 0.1, self.psser + 0.1)

Review comment:
       Arithmetic operations between`psser` and `psser.astype(int)` are tested.
   
   `psser + psser` will raise `TypeError`, while pandas treat it as `__and__`. A follow-up PR might be needed to support it. Would you prefer it to be implemented in this PR?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847442932


   **[Test build #138895 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138895/testReport)** for PR 32611 at commit [`203f90a`](https://github.com/apache/spark/commit/203f90aab81cecf2591f2fe2aa4b5466b2e486ae).
    * This patch **fails Python style tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847391398


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43410/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850542913


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139066/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [WIP][SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845425762


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43285/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845618643


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138785/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847419100


   **[Test build #138891 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138891/testReport)** for PR 32611 at commit [`7ea2876`](https://github.com/apache/spark/commit/7ea2876d0c5083ffa2eddbc634edab6e4a74d71a).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846130825


   **[Test build #138811 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138811/testReport)** for PR 32611 at commit [`bd63058`](https://github.com/apache/spark/commit/bd630589cc6d686315053e6024cc61f4fb396bfd).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845635027


   **[Test build #138788 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138788/testReport)** for PR 32611 at commit [`e626c52`](https://github.com/apache/spark/commit/e626c52245521fdeb0ee46c68950605f87a987bd).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-848059344


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43457/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849205409


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43510/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845557983


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43292/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845534978


   **[Test build #138769 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138769/testReport)** for PR 32611 at commit [`b0b2a3b`](https://github.com/apache/spark/commit/b0b2a3bb61680720bb2c1be86c87a2f8dfa847d9).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845635353


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138788/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846317781


   **[Test build #138818 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138818/testReport)** for PR 32611 at commit [`925e34b`](https://github.com/apache/spark/commit/925e34b3bffb6496d62425fa08d36177589cb17d).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] ueshin commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r640957750



##########
File path: python/pyspark/pandas/tests/test_dataframe.py
##########
@@ -121,7 +121,7 @@ def test_extension_dtypes(self):
         psdf = ps.from_pandas(pdf)
 
         self._check_extension(psdf, pdf)
-        self._check_extension(psdf + F.lit(1).cast("byte"), pdf + 1)
+        # self._check_extension(psdf + F.lit(1).cast("byte"), pdf + 1)

Review comment:
       Yes, shall we just remove it?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847245881


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43402/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846328532


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43340/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845534978


   **[Test build #138769 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138769/testReport)** for PR 32611 at commit [`b0b2a3b`](https://github.com/apache/spark/commit/b0b2a3bb61680720bb2c1be86c87a2f8dfa847d9).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847365650


   **[Test build #138888 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138888/testReport)** for PR 32611 at commit [`99e66a1`](https://github.com/apache/spark/commit/99e66a18a07f0ba69a0c036b36374dd8be74ba49).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847435504


   Kubernetes integration test unable to build dist.
   
   exiting with code: 1
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43413/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847240922


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138880/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845707439


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43311/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845556717


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43291/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846081872


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138809/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849985493


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43551/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845635337


   **[Test build #138788 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138788/testReport)** for PR 32611 at commit [`e626c52`](https://github.com/apache/spark/commit/e626c52245521fdeb0ee46c68950605f87a987bd).
    * This patch **fails Python style tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850707017


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139073/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON]Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845511826


   **[Test build #138768 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138768/testReport)** for PR 32611 at commit [`6c5a5cf`](https://github.com/apache/spark/commit/6c5a5cf431e0e23b67fdcbee578563dffb366947).
    * This patch **fails Python style tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850690763


   **[Test build #139073 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139073/testReport)** for PR 32611 at commit [`db12fb1`](https://github.com/apache/spark/commit/db12fb1c854051b48caf6a253ef6dcb70694db51).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r638137564



##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +46,183 @@ class BooleanOps(DataTypeOps):
     @property
     def pretty_name(self) -> str:
         return 'booleans'
+
+    def add(self, left, right) -> Union["Series", "Index"]:
+        if isinstance(right, numbers.Number):
+            left = left.spark.transform(lambda scol: scol.cast(as_spark_type(type(right))))
+            return column_op(Column.__add__)(left, right)

Review comment:
       Makes sense, addition and other forward operations are applied this approach.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847441076






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-846205753


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43333/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849187308


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138990/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847442945


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138895/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849188150


   **[Test build #138991 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138991/testReport)** for PR 32611 at commit [`5905012`](https://github.com/apache/spark/commit/5905012338a16ba500094b9cae33bb783e86faff).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849957217


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43551/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r636607352



##########
File path: python/pyspark/pandas/data_type_ops/num_ops.py
##########
@@ -16,31 +16,46 @@
 #
 
 import numbers
-from typing import TYPE_CHECKING, Union
+from typing import Any, TYPE_CHECKING, Union
 
 import numpy as np
 from pandas.api.types import CategoricalDtype
 
 from pyspark.sql import Column, functions as F
 from pyspark.sql.types import (
+    BooleanType,
     NumericType,
     StringType,
     TimestampType,
 )
 
 from pyspark.pandas.base import column_op, IndexOpsMixin, numpy_column_op
-from pyspark.pandas.data_type_ops.base import DataTypeOps
+from pyspark.pandas.data_type_ops.base import DataTypeOps, transform_boolean_operand_to_numeric
 from pyspark.pandas.spark import functions as SF
+from pyspark.sql.column import Column
+
 
 if TYPE_CHECKING:
     from pyspark.pandas.indexes import Index  # noqa: F401 (SPARK-34943)
     from pyspark.pandas.series import Series  # noqa: F401 (SPARK-34943)
 
 
+def is_valid_operand_for_numeric_arithmetic(operand: Any) -> bool:
+    """Check whether the operand is valid for arithmetic operations against numerics."""
+    if isinstance(operand, numbers.Number):
+        return True
+    elif isinstance(operand, IndexOpsMixin):
+        if isinstance(operand.dtype, CategoricalDtype):
+            return False
+        else:
+            return isinstance(operand.spark.data_type, NumericType) or isinstance(
+                operand.spark.data_type, BooleanType)
+    else:
+        return isinstance(operand, Column)

Review comment:
       Returning True when the operand is a Column to cover the cases as below:
   
   `left.astype("long") - F.lit(right).cast(as_spark_type("long"))` (when a datatime series - a datatime)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-849933905


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139033/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850569990


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43587/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-850569990


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43587/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847240922


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138880/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-databricks commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r637305932



##########
File path: python/pyspark/pandas/tests/data_type_ops/test_num_ops.py
##########
@@ -30,6 +34,14 @@ class NumOpsTest(PandasOnSparkTestCase, TestCasesUtils):
     returns float32.
     The underlying reason is the respective Spark operations return DoubleType always.
     """
+    @property
+    def float_pser(self):
+        return pd.Series([1, 2, 3], dtype=float)
+
+    @property
+    def float_psser(self):
+        return ps.from_pandas(self.float_pser)
+
     def test_add(self):
         for pser, psser in self.numeric_pser_psser_pairs:
             self.assert_eq(pser + pser, psser + psser)

Review comment:
       Arithmetic operations involving `bool` are not supported now. TypeErrors are introduced for these cases for now.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847417614






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-845635353


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138788/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847442945


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138895/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] ueshin commented on a change in pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #32611:
URL: https://github.com/apache/spark/pull/32611#discussion_r639154215



##########
File path: python/pyspark/pandas/data_type_ops/boolean_ops.py
##########
@@ -26,3 +39,144 @@ class BooleanOps(DataTypeOps):
     @property
     def pretty_name(self) -> str:
         return 'booleans'
+
+    def add(self, left, right) -> Union["Series", "Index"]:
+        if not is_valid_operand_for_numeric_arithmetic(right, allow_bool=False):
+            raise TypeError(
+                "Addition can not be applied to %s and the given type." % self.pretty_name)
+
+        if isinstance(right, numbers.Number) and not isinstance(right, bool):
+            left = left.spark.transform(lambda scol: scol.cast(as_spark_type(type(right))))
+            return left + right
+        else:
+            assert isinstance(right, IndexOpsMixin)

Review comment:
       I guess `right` of type `spark.Column` could fall into this path.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32611: [SPARK-35314][PYTHON] Support arithmetic operations against bool IndexOpsMixin

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32611:
URL: https://github.com/apache/spark/pull/32611#issuecomment-847455951


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43417/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org