You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/07/29 20:00:21 UTC
[GitHub] [spark] xinrong-databricks opened a new pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
xinrong-databricks opened a new pull request #33581:
URL: https://github.com/apache/spark/pull/33581
### What changes were proposed in this pull request?
Better error messages for DataTypeOps against lists
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
Yes. A TypeError message will be showed rather than a Spark exception.
### How was this patch tested?
Unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #33581:
URL: https://github.com/apache/spark/pull/33581#discussion_r681160066
##########
File path: python/pyspark/pandas/data_type_ops/base.py
##########
@@ -188,6 +188,12 @@ def _as_other_type(
return index_ops._with_new_scol(scol, field=InternalField(dtype=dtype))
+def _sanitize_list_like(operand: Any) -> None:
Review comment:
Python's type annotations are just a hint for users, but they can still pass any objects, unlike Scala/Java.
I think type checking and showing a proper message are our responsibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33581:
URL: https://github.com/apache/spark/pull/33581#issuecomment-889437392
**[Test build #141855 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141855/testReport)** for PR 33581 at commit [`af93e9c`](https://github.com/apache/spark/commit/af93e9c0912eca8e175138ad175f3909241b19c3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #33581:
URL: https://github.com/apache/spark/pull/33581#discussion_r681451616
##########
File path: python/pyspark/pandas/data_type_ops/base.py
##########
@@ -314,9 +320,11 @@ def __or__(self, left: IndexOpsLike, right: Any) -> SeriesOrIndex:
raise TypeError("Bitwise or can not be applied to %s." % self.pretty_name)
def rand(self, left: IndexOpsLike, right: Any) -> SeriesOrIndex:
Review comment:
Oh, yeah I got it. Looks good to me as is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33581:
URL: https://github.com/apache/spark/pull/33581#issuecomment-890226172
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46418/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33581:
URL: https://github.com/apache/spark/pull/33581#issuecomment-889526065
**[Test build #141859 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141859/testReport)** for PR 33581 at commit [`89e9128`](https://github.com/apache/spark/commit/89e912842f0530c2e0fe1312c039641b781f5ba9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33581:
URL: https://github.com/apache/spark/pull/33581#issuecomment-889474067
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141855/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on pull request #33581:
URL: https://github.com/apache/spark/pull/33581#issuecomment-891292376
@ueshin Certainly, added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33581:
URL: https://github.com/apache/spark/pull/33581#issuecomment-890166784
**[Test build #141909 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141909/testReport)** for PR 33581 at commit [`b2f2878`](https://github.com/apache/spark/commit/b2f28784bfd96f842c3e44f3caf4e1f23ef39d87).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33581:
URL: https://github.com/apache/spark/pull/33581#issuecomment-891604843
Merged to master and branch-3.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on pull request #33581:
URL: https://github.com/apache/spark/pull/33581#issuecomment-890224693
CC @ueshin @HyukjinKwon @itholic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
ueshin commented on pull request #33581:
URL: https://github.com/apache/spark/pull/33581#issuecomment-891226323
@xinrong-databricks Could you add the difference of error messages before and after this PR to clarify?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #33581:
URL: https://github.com/apache/spark/pull/33581#discussion_r681160066
##########
File path: python/pyspark/pandas/data_type_ops/base.py
##########
@@ -188,6 +188,12 @@ def _as_other_type(
return index_ops._with_new_scol(scol, field=InternalField(dtype=dtype))
+def _sanitize_list_like(operand: Any) -> None:
Review comment:
Python's type annotations are just a hint for users, but they can still pass any objects, unlike Scala/Java.
I think we should check the actual type and show a proper message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33581:
URL: https://github.com/apache/spark/pull/33581#issuecomment-889544314
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #33581:
URL: https://github.com/apache/spark/pull/33581#discussion_r680639379
##########
File path: python/pyspark/pandas/data_type_ops/base.py
##########
@@ -314,9 +320,11 @@ def __or__(self, left: IndexOpsLike, right: Any) -> SeriesOrIndex:
raise TypeError("Bitwise or can not be applied to %s." % self.pretty_name)
def rand(self, left: IndexOpsLike, right: Any) -> SeriesOrIndex:
Review comment:
As @HyukjinKwon mentioned in https://github.com/apache/spark/pull/33581/files#r680433058,
I think maybe we can just fix the typing from `right: Any` to something like `right: Union[List, Tuple, Dict, Set]` rather adding function ?? (Or we can take both typing and function ??)
I have no string opinion, too, but anyway the typing here `right: Any` seems wrong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #33581:
URL: https://github.com/apache/spark/pull/33581
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #33581:
URL: https://github.com/apache/spark/pull/33581#discussion_r681167073
##########
File path: python/pyspark/pandas/data_type_ops/base.py
##########
@@ -314,9 +320,11 @@ def __or__(self, left: IndexOpsLike, right: Any) -> SeriesOrIndex:
raise TypeError("Bitwise or can not be applied to %s." % self.pretty_name)
def rand(self, left: IndexOpsLike, right: Any) -> SeriesOrIndex:
Review comment:
We disallow `List, Tuple, Dict, Set` for `right`. The function added is to raise a TypeError for these list-likes. Sorry if there is any confusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #33581:
URL: https://github.com/apache/spark/pull/33581#discussion_r680639379
##########
File path: python/pyspark/pandas/data_type_ops/base.py
##########
@@ -314,9 +320,11 @@ def __or__(self, left: IndexOpsLike, right: Any) -> SeriesOrIndex:
raise TypeError("Bitwise or can not be applied to %s." % self.pretty_name)
def rand(self, left: IndexOpsLike, right: Any) -> SeriesOrIndex:
Review comment:
As @HyukjinKwon mentioned in https://github.com/apache/spark/pull/33581#discussion_r680433058,
I think maybe we can just fix the typing from `right: Any` to something like `right: Union[List, Tuple, Dict, Set]` rather adding function ?? (Or we can take both typing and function ??)
I have no string opinion, too, but anyway the typing here `right: Any` seems not correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33581:
URL: https://github.com/apache/spark/pull/33581#issuecomment-890226163
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46418/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33581:
URL: https://github.com/apache/spark/pull/33581#issuecomment-889437392
**[Test build #141855 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141855/testReport)** for PR 33581 at commit [`af93e9c`](https://github.com/apache/spark/commit/af93e9c0912eca8e175138ad175f3909241b19c3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33581:
URL: https://github.com/apache/spark/pull/33581#issuecomment-890221108
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141909/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33581:
URL: https://github.com/apache/spark/pull/33581#issuecomment-890216141
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46418/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33581:
URL: https://github.com/apache/spark/pull/33581#issuecomment-890166784
**[Test build #141909 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141909/testReport)** for PR 33581 at commit [`b2f2878`](https://github.com/apache/spark/commit/b2f28784bfd96f842c3e44f3caf4e1f23ef39d87).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33581:
URL: https://github.com/apache/spark/pull/33581#issuecomment-889502949
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46366/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33581:
URL: https://github.com/apache/spark/pull/33581#discussion_r680433058
##########
File path: python/pyspark/pandas/data_type_ops/base.py
##########
@@ -188,6 +188,12 @@ def _as_other_type(
return index_ops._with_new_scol(scol, field=InternalField(dtype=dtype))
+def _sanitize_list_like(operand: Any) -> None:
Review comment:
I wonder if we should add them actually since we're typing explicitly which enables static analysis. No strong opinion tho.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33581:
URL: https://github.com/apache/spark/pull/33581#issuecomment-889502949
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46366/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #33581:
URL: https://github.com/apache/spark/pull/33581#discussion_r680639379
##########
File path: python/pyspark/pandas/data_type_ops/base.py
##########
@@ -314,9 +320,11 @@ def __or__(self, left: IndexOpsLike, right: Any) -> SeriesOrIndex:
raise TypeError("Bitwise or can not be applied to %s." % self.pretty_name)
def rand(self, left: IndexOpsLike, right: Any) -> SeriesOrIndex:
Review comment:
As @HyukjinKwon mentioned in https://github.com/apache/spark/pull/33581#discussion_r680433058,
I think maybe we can just fix the typing from `right: Any` to something like `right: Union[List, Tuple, Dict, Set]` rather adding function ?? (Or we can take both typing and function ??)
I have no string opinion, too, but anyway the typing here `right: Any` seems not correct if we cannot accept the Python list-like objects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33581:
URL: https://github.com/apache/spark/pull/33581#issuecomment-890221108
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141909/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33581:
URL: https://github.com/apache/spark/pull/33581#issuecomment-889539485
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46370/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33581:
URL: https://github.com/apache/spark/pull/33581#issuecomment-889534287
**[Test build #141859 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141859/testReport)** for PR 33581 at commit [`89e9128`](https://github.com/apache/spark/commit/89e912842f0530c2e0fe1312c039641b781f5ba9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33581:
URL: https://github.com/apache/spark/pull/33581#issuecomment-890190903
**[Test build #141909 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141909/testReport)** for PR 33581 at commit [`b2f2878`](https://github.com/apache/spark/commit/b2f28784bfd96f842c3e44f3caf4e1f23ef39d87).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33581:
URL: https://github.com/apache/spark/pull/33581#issuecomment-889468810
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46366/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33581:
URL: https://github.com/apache/spark/pull/33581#issuecomment-889474067
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141855/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33581:
URL: https://github.com/apache/spark/pull/33581#issuecomment-889487804
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46366/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33581:
URL: https://github.com/apache/spark/pull/33581#issuecomment-889526065
**[Test build #141859 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141859/testReport)** for PR 33581 at commit [`89e9128`](https://github.com/apache/spark/commit/89e912842f0530c2e0fe1312c039641b781f5ba9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33581:
URL: https://github.com/apache/spark/pull/33581#issuecomment-890226172
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46418/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33581:
URL: https://github.com/apache/spark/pull/33581#issuecomment-889544314
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33581:
URL: https://github.com/apache/spark/pull/33581#issuecomment-889454678
**[Test build #141855 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141855/testReport)** for PR 33581 at commit [`af93e9c`](https://github.com/apache/spark/commit/af93e9c0912eca8e175138ad175f3909241b19c3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #33581: [SPARK-36192][PYTHON] Better error messages for DataTypeOps against lists
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #33581:
URL: https://github.com/apache/spark/pull/33581#discussion_r680639379
##########
File path: python/pyspark/pandas/data_type_ops/base.py
##########
@@ -314,9 +320,11 @@ def __or__(self, left: IndexOpsLike, right: Any) -> SeriesOrIndex:
raise TypeError("Bitwise or can not be applied to %s." % self.pretty_name)
def rand(self, left: IndexOpsLike, right: Any) -> SeriesOrIndex:
Review comment:
As @HyukjinKwon mentioned in https://github.com/apache/spark/pull/33581#discussion_r680433058,
I think maybe we can just fix the typing from `right: Any` to something like `right: Union[List, Tuple, Dict, Set]` rather adding function ?? (Or we can take both typing and function ??)
I have no string opinion, too, but anyway the typing here `Any` seems not correct if we cannot accept the list-like Python objects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org