You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/10/18 22:58:18 UTC
[GitHub] [spark] ueshin opened a new pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
ueshin opened a new pull request #34318:
URL: https://github.com/apache/spark/pull/34318
### What changes were proposed in this pull request?
Cleans up inlining type hints under SQL module.
### Why are the changes needed?
Now that most of type hits under the SQL module are inlined, we should clean up for the module now.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
`lint-python` and existing tests should pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-947002755
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48894/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #34318:
URL: https://github.com/apache/spark/pull/34318#discussion_r733151730
##########
File path: python/pyspark/sql/observation.py
##########
@@ -14,17 +14,16 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-from typing import Any, Dict, Optional, TYPE_CHECKING
+from typing import Any, Dict, Optional
+
+from py4j.java_gateway import JavaObject, JVMView
Review comment:
Sure, then let's have the ignores for now. I'll update soon. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
dchvn commented on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-946354661
I think some other places may be cleaned up too. Thank you!
https://github.com/ueshin/apache-spark/blob/2f175be8acb44aa793519fc8f50b2cc3830350a6/python/pyspark/sql/streaming.py#L322
https://github.com/ueshin/apache-spark/blob/2f175be8acb44aa793519fc8f50b2cc3830350a6/python/pyspark/sql/dataframe.py#L109
https://github.com/ueshin/apache-spark/blob/2f175be8acb44aa793519fc8f50b2cc3830350a6/python/pyspark/sql/dataframe.py#L1280
https://github.com/ueshin/apache-spark/blob/2f175be8acb44aa793519fc8f50b2cc3830350a6/python/pyspark/sql/dataframe.py#L1654
https://github.com/ueshin/apache-spark/blob/2f175be8acb44aa793519fc8f50b2cc3830350a6/python/pyspark/sql/dataframe.py#L1658
https://github.com/ueshin/apache-spark/blob/2f175be8acb44aa793519fc8f50b2cc3830350a6/python/pyspark/sql/context.py#L37
https://github.com/ueshin/apache-spark/blob/2f175be8acb44aa793519fc8f50b2cc3830350a6/python/pyspark/sql/context.py#L124
https://github.com/ueshin/apache-spark/blob/2f175be8acb44aa793519fc8f50b2cc3830350a6/python/pyspark/sql/context.py#L730
https://github.com/ueshin/apache-spark/blob/2f175be8acb44aa793519fc8f50b2cc3830350a6/python/pyspark/sql/readwriter.py#L362
https://github.com/ueshin/apache-spark/blob/2f175be8acb44aa793519fc8f50b2cc3830350a6/python/pyspark/sql/readwriter.py#L575
https://github.com/ueshin/apache-spark/blob/2f175be8acb44aa793519fc8f50b2cc3830350a6/python/pyspark/sql/readwriter.py#L787
https://github.com/ueshin/apache-spark/blob/2f175be8acb44aa793519fc8f50b2cc3830350a6/python/pyspark/sql/readwriter.py#L852
https://github.com/ueshin/apache-spark/blob/2f175be8acb44aa793519fc8f50b2cc3830350a6/python/pyspark/sql/readwriter.py#L902
https://github.com/ueshin/apache-spark/blob/2f175be8acb44aa793519fc8f50b2cc3830350a6/python/pyspark/sql/readwriter.py#L1389
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34318:
URL: https://github.com/apache/spark/pull/34318#discussion_r733148023
##########
File path: python/pyspark/sql/observation.py
##########
@@ -14,17 +14,16 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-from typing import Any, Dict, Optional, TYPE_CHECKING
+from typing import Any, Dict, Optional
+
+from py4j.java_gateway import JavaObject, JVMView
Review comment:
Personally, I have a preference for adding ignores to imports that have no stubs or built-in annotations, because it clearly indicates that we don't expect these things to typecheck. And these take precedence over external config, so I can easily experiment with different options. But I am fine with either approach, as long as we are consistent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-947924886
Thanks, LGTM!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-946949913
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144421/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
ueshin commented on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-948105766
Thanks! merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-948090661
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144474/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #34318:
URL: https://github.com/apache/spark/pull/34318#discussion_r733164013
##########
File path: python/pyspark/pandas/data_type_ops/base.py
##########
@@ -395,7 +395,7 @@ def eq(self, left: IndexOpsLike, right: Any) -> SeriesOrIndex:
collected_structed_scol = F.collect_list(structed_scol)
# Sort the array by NATURAL_ORDER_COLUMN so that we can guarantee the order.
collected_structed_scol = F.array_sort(collected_structed_scol)
- right_values_scol = F.array([F.lit(x) for x in right]) # type: ignore
+ right_values_scol = F.array(*(F.lit(x) for x in right))
Review comment:
Maybe we should discuss it in a separate PR because we also need to discuss `struct` or something that has a similar overloads, at the same time or under another umbrella ticket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-947002755
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48894/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-946248311
**[Test build #144382 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144382/testReport)** for PR 34318 at commit [`2f175be`](https://github.com/apache/spark/commit/2f175be8acb44aa793519fc8f50b2cc3830350a6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #34318:
URL: https://github.com/apache/spark/pull/34318#discussion_r733142511
##########
File path: python/pyspark/sql/observation.py
##########
@@ -14,17 +14,16 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-from typing import Any, Dict, Optional, TYPE_CHECKING
+from typing import Any, Dict, Optional
+
+from py4j.java_gateway import JavaObject, JVMView
Review comment:
WDYT, @zero323 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #34318:
URL: https://github.com/apache/spark/pull/34318#discussion_r732034743
##########
File path: python/pyspark/sql/observation.py
##########
@@ -14,17 +14,16 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-from typing import Any, Dict, Optional, TYPE_CHECKING
+from typing import Any, Dict, Optional
+
+from py4j.java_gateway import JavaObject, JVMView
Review comment:
Seems not. Is it because there is an entry for `py4j` in `mypy.ini`?
```
[mypy-py4j.*]
ignore_missing_imports = True
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-948090661
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144474/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34318:
URL: https://github.com/apache/spark/pull/34318#discussion_r733153779
##########
File path: python/pyspark/pandas/data_type_ops/base.py
##########
@@ -395,7 +395,7 @@ def eq(self, left: IndexOpsLike, right: Any) -> SeriesOrIndex:
collected_structed_scol = F.collect_list(structed_scol)
# Sort the array by NATURAL_ORDER_COLUMN so that we can guarantee the order.
collected_structed_scol = F.array_sort(collected_structed_scol)
- right_values_scol = F.array([F.lit(x) for x in right]) # type: ignore
+ right_values_scol = F.array(*(F.lit(x) for x in right))
Review comment:
Seems like we should add an overload to `array`, right?
```python
@overload
def array(cols: Union[List["ColumnOrName"], Tuple[ "ColumnOrName", ...]]) -> Column:
...
@overload
def array(*cols: "ColumnOrName") -> Column:
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
ueshin commented on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-948105702
I'd merge this now. Feel free to leave more comments if any, then I'll address them in a follow-up PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-948058279
**[Test build #144474 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144474/testReport)** for PR 34318 at commit [`3a197c2`](https://github.com/apache/spark/commit/3a197c277d68d309541f346ad983c385064fff71).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-946257799
**[Test build #144382 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144382/testReport)** for PR 34318 at commit [`2f175be`](https://github.com/apache/spark/commit/2f175be8acb44aa793519fc8f50b2cc3830350a6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
ueshin commented on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-946231349
cc @zero323 @dchvn @xinrong-databricks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-946951510
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48894/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-946264624
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48856/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-946282635
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48856/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-946269918
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144382/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-946933131
**[Test build #144421 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144421/testReport)** for PR 34318 at commit [`9444676`](https://github.com/apache/spark/commit/944467634275a5c5a98dc6eebcf3b48359c05d99).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #34318:
URL: https://github.com/apache/spark/pull/34318#discussion_r733038619
##########
File path: python/pyspark/sql/observation.py
##########
@@ -14,17 +14,16 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-from typing import Any, Dict, Optional, TYPE_CHECKING
+from typing import Any, Dict, Optional
+
+from py4j.java_gateway import JavaObject, JVMView
Review comment:
I am still confused why some `from py4j.java_gateway import JavaObject` needs `# type: ignore[import]` though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
ueshin commented on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-947870882
also cc @HyukjinKwon @itholic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
dchvn commented on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-947224547
LGTM. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34318:
URL: https://github.com/apache/spark/pull/34318#discussion_r733153779
##########
File path: python/pyspark/pandas/data_type_ops/base.py
##########
@@ -395,7 +395,7 @@ def eq(self, left: IndexOpsLike, right: Any) -> SeriesOrIndex:
collected_structed_scol = F.collect_list(structed_scol)
# Sort the array by NATURAL_ORDER_COLUMN so that we can guarantee the order.
collected_structed_scol = F.array_sort(collected_structed_scol)
- right_values_scol = F.array([F.lit(x) for x in right]) # type: ignore
+ right_values_scol = F.array(*(F.lit(x) for x in right))
Review comment:
Seems like we should add an overload to `array`, right?
```python
@overload
def array(cols: Union[List["ColumnOrName"], Tuple["ColumnOrName", ...]]) -> Column:
...
@overload
def array(*cols: "ColumnOrName") -> Column:
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #34318:
URL: https://github.com/apache/spark/pull/34318#discussion_r733164013
##########
File path: python/pyspark/pandas/data_type_ops/base.py
##########
@@ -395,7 +395,7 @@ def eq(self, left: IndexOpsLike, right: Any) -> SeriesOrIndex:
collected_structed_scol = F.collect_list(structed_scol)
# Sort the array by NATURAL_ORDER_COLUMN so that we can guarantee the order.
collected_structed_scol = F.array_sort(collected_structed_scol)
- right_values_scol = F.array([F.lit(x) for x in right]) # type: ignore
+ right_values_scol = F.array(*(F.lit(x) for x in right))
Review comment:
Maybe we should discuss it in a separate PR because we also need to discuss `struct` or something that has a similar overloads, at the same time or under another umbrella ticket.
I'm afraid I don't think we should provide the signature, though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-946905209
**[Test build #144421 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144421/testReport)** for PR 34318 at commit [`9444676`](https://github.com/apache/spark/commit/944467634275a5c5a98dc6eebcf3b48359c05d99).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-946905209
**[Test build #144421 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144421/testReport)** for PR 34318 at commit [`9444676`](https://github.com/apache/spark/commit/944467634275a5c5a98dc6eebcf3b48359c05d99).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-948110344
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48947/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-946949913
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144421/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #34318:
URL: https://github.com/apache/spark/pull/34318#discussion_r733164013
##########
File path: python/pyspark/pandas/data_type_ops/base.py
##########
@@ -395,7 +395,7 @@ def eq(self, left: IndexOpsLike, right: Any) -> SeriesOrIndex:
collected_structed_scol = F.collect_list(structed_scol)
# Sort the array by NATURAL_ORDER_COLUMN so that we can guarantee the order.
collected_structed_scol = F.array_sort(collected_structed_scol)
- right_values_scol = F.array([F.lit(x) for x in right]) # type: ignore
+ right_values_scol = F.array(*(F.lit(x) for x in right))
Review comment:
Maybe we should discuss it in separate PRs because we also need to discuss `struct` or something that has a similar overloads at the same time or another umbrella ticket.
I'm afraid I think we shouldn't provide the signature, though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-948084617
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48947/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-948075581
**[Test build #144474 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144474/testReport)** for PR 34318 at commit [`3a197c2`](https://github.com/apache/spark/commit/3a197c277d68d309541f346ad983c385064fff71).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-946269918
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144382/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-946294158
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48856/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-946248311
**[Test build #144382 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144382/testReport)** for PR 34318 at commit [`2f175be`](https://github.com/apache/spark/commit/2f175be8acb44aa793519fc8f50b2cc3830350a6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-946986870
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48894/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-946294158
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48856/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34318:
URL: https://github.com/apache/spark/pull/34318#discussion_r731725016
##########
File path: python/pyspark/sql/observation.py
##########
@@ -14,17 +14,16 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-from typing import Any, Dict, Optional, TYPE_CHECKING
+from typing import Any, Dict, Optional
+
+from py4j.java_gateway import JavaObject, JVMView
Review comment:
Don't we need ignore here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-948115873
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48947/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-948115873
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48947/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34318:
URL: https://github.com/apache/spark/pull/34318#issuecomment-948058279
**[Test build #144474 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144474/testReport)** for PR 34318 at commit [`3a197c2`](https://github.com/apache/spark/commit/3a197c277d68d309541f346ad983c385064fff71).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #34318:
URL: https://github.com/apache/spark/pull/34318#discussion_r733164013
##########
File path: python/pyspark/pandas/data_type_ops/base.py
##########
@@ -395,7 +395,7 @@ def eq(self, left: IndexOpsLike, right: Any) -> SeriesOrIndex:
collected_structed_scol = F.collect_list(structed_scol)
# Sort the array by NATURAL_ORDER_COLUMN so that we can guarantee the order.
collected_structed_scol = F.array_sort(collected_structed_scol)
- right_values_scol = F.array([F.lit(x) for x in right]) # type: ignore
+ right_values_scol = F.array(*(F.lit(x) for x in right))
Review comment:
Maybe we should discuss it in a separate PR because we also need to discuss `struct` or something that has a similar overloads at the same time or another umbrella ticket.
I'm afraid I think we shouldn't provide the signature, though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #34318:
URL: https://github.com/apache/spark/pull/34318#discussion_r733164013
##########
File path: python/pyspark/pandas/data_type_ops/base.py
##########
@@ -395,7 +395,7 @@ def eq(self, left: IndexOpsLike, right: Any) -> SeriesOrIndex:
collected_structed_scol = F.collect_list(structed_scol)
# Sort the array by NATURAL_ORDER_COLUMN so that we can guarantee the order.
collected_structed_scol = F.array_sort(collected_structed_scol)
- right_values_scol = F.array([F.lit(x) for x in right]) # type: ignore
+ right_values_scol = F.array(*(F.lit(x) for x in right))
Review comment:
Maybe we should discuss it in a separate PR because we also need to discuss `struct` or something that has a similar overloads, at the same time or under another umbrella ticket.
I'm afraid I think we shouldn't provide the signature, though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #34318:
URL: https://github.com/apache/spark/pull/34318#discussion_r733130854
##########
File path: python/pyspark/sql/observation.py
##########
@@ -14,17 +14,16 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-from typing import Any, Dict, Optional, TYPE_CHECKING
+from typing import Any, Dict, Optional
+
+from py4j.java_gateway import JavaObject, JVMView
Review comment:
Actually I don't think we need all of them now because of the `mypy.ini` entry. I haven't tried, though.
We can try it in a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin closed pull request #34318: [SPARK-37048][PYTHON] Clean up inlining type hints under SQL module
Posted by GitBox <gi...@apache.org>.
ueshin closed pull request #34318:
URL: https://github.com/apache/spark/pull/34318
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org