You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/11/12 01:29:33 UTC
[GitHub] [spark] ueshin opened a new pull request #34563: [SPARK-37296][PYTHON] Add missing type hints in python/pyspark/util.py
ueshin opened a new pull request #34563:
URL: https://github.com/apache/spark/pull/34563
### What changes were proposed in this pull request?
Adds missing type hints in `python/pyspark/util.py`.
### Why are the changes needed?
Some of type hints in `python/pyspark/util.py` are missing, but the error was ignored by setting `disallow_untyped_defs = False` in `mypy.ini`. We should remove the setting and add the missing type hints.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
`lint-python` should pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34563: [SPARK-37296][PYTHON] Add missing type hints in python/pyspark/util.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34563:
URL: https://github.com/apache/spark/pull/34563#issuecomment-966761993
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145128/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34563: [SPARK-37296][PYTHON] Add missing type hints in python/pyspark/util.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34563:
URL: https://github.com/apache/spark/pull/34563#issuecomment-966804907
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49599/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34563: [SPARK-37296][PYTHON] Add missing type hints in python/pyspark/util.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34563:
URL: https://github.com/apache/spark/pull/34563#issuecomment-966804907
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49599/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34563: [SPARK-37296][PYTHON] Add missing type hints in python/pyspark/util.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34563:
URL: https://github.com/apache/spark/pull/34563#issuecomment-966761993
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145128/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34563: [SPARK-37296][PYTHON] Add missing type hints in python/pyspark/util.py
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34563:
URL: https://github.com/apache/spark/pull/34563#issuecomment-966796672
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34563: [SPARK-37296][PYTHON] Add missing type hints in python/pyspark/util.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34563:
URL: https://github.com/apache/spark/pull/34563#issuecomment-966745880
**[Test build #145128 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145128/testReport)** for PR 34563 at commit [`8505fba`](https://github.com/apache/spark/commit/8505fbae8faa605b22c27e79cb7793e98962af2f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34563: [SPARK-37296][PYTHON] Add missing type hints in python/pyspark/util.py
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34563:
URL: https://github.com/apache/spark/pull/34563#discussion_r747925755
##########
File path: python/pyspark/util.py
##########
@@ -369,39 +367,49 @@ class InheritableThread(threading.Thread):
This API is experimental.
"""
- def __init__(self, target, *args, **kwargs):
+ def __init__(self, target: Callable, *args: Any, **kwargs: Any):
Review comment:
Yeah. I will give a shot to deprecate this API by fixing it in Py4J in Apache Spark 3.3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34563: [SPARK-37296][PYTHON] Add missing type hints in python/pyspark/util.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34563:
URL: https://github.com/apache/spark/pull/34563#issuecomment-966756708
**[Test build #145128 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145128/testReport)** for PR 34563 at commit [`8505fba`](https://github.com/apache/spark/commit/8505fbae8faa605b22c27e79cb7793e98962af2f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34563: [SPARK-37296][PYTHON] Add missing type hints in python/pyspark/util.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34563:
URL: https://github.com/apache/spark/pull/34563#issuecomment-966745880
**[Test build #145128 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145128/testReport)** for PR 34563 at commit [`8505fba`](https://github.com/apache/spark/commit/8505fbae8faa605b22c27e79cb7793e98962af2f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #34563: [SPARK-37296][PYTHON] Add missing type hints in python/pyspark/util.py
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #34563:
URL: https://github.com/apache/spark/pull/34563
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #34563: [SPARK-37296][PYTHON] Add missing type hints in python/pyspark/util.py
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #34563:
URL: https://github.com/apache/spark/pull/34563#discussion_r747922607
##########
File path: python/pyspark/util.py
##########
@@ -369,39 +367,49 @@ class InheritableThread(threading.Thread):
This API is experimental.
"""
- def __init__(self, target, *args, **kwargs):
+ def __init__(self, target: Callable, *args: Any, **kwargs: Any):
Review comment:
@HyukjinKwon This signature might not be good because this is different from the signature of `Thread.__init__`.
```py
def __init__(
self,
group: None = ...,
target: Optional[Callable[..., Any]] = ...,
name: Optional[str] = ...,
args: Iterable[Any] = ...,
kwargs: Optional[Mapping[str, Any]] = ...,
*,
daemon: Optional[bool] = ...,
) -> None: ...
```
I guess users can't specify `group`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34563: [SPARK-37296][PYTHON] Add missing type hints in python/pyspark/util.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34563:
URL: https://github.com/apache/spark/pull/34563#issuecomment-966780861
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49599/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34563: [SPARK-37296][PYTHON] Add missing type hints in python/pyspark/util.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34563:
URL: https://github.com/apache/spark/pull/34563#issuecomment-966796537
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49599/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #34563: [SPARK-37296][PYTHON] Add missing type hints in python/pyspark/util.py
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #34563:
URL: https://github.com/apache/spark/pull/34563#discussion_r747923851
##########
File path: python/pyspark/util.py
##########
@@ -369,39 +367,49 @@ class InheritableThread(threading.Thread):
This API is experimental.
"""
- def __init__(self, target, *args, **kwargs):
+ def __init__(self, target: Callable, *args: Any, **kwargs: Any):
from pyspark import SparkContext
- if isinstance(SparkContext._gateway, ClientServer):
+ if isinstance(SparkContext._gateway, ClientServer): # type: ignore[attr-defined]
# Here's when the pinned-thread mode (PYSPARK_PIN_THREAD) is on.
- def copy_local_properties(*a, **k):
+ def copy_local_properties(*a: Any, **k: Any) -> Any:
# self._props is set before starting the thread to match the behavior with JVM.
assert hasattr(self, "_props")
- SparkContext._active_spark_context._jsc.sc().setLocalProperties(self._props)
+ SparkContext._active_spark_context._jsc.sc().setLocalProperties( # type: ignore[attr-defined]
+ self._props
+ )
try:
return target(*a, **k)
finally:
InheritableThread._clean_py4j_conn_for_current_thread()
- super(InheritableThread, self).__init__(target=copy_local_properties, *args, **kwargs)
+ super(InheritableThread, self).__init__(
+ target=copy_local_properties, *args, **kwargs # type: ignore[misc]
+ )
else:
- super(InheritableThread, self).__init__(target=target, *args, **kwargs)
+ super(InheritableThread, self).__init__(
+ target=target, *args, **kwargs # type: ignore[misc]
+ )
Review comment:
```
python/pyspark/util.py:390: error: "__init__" of "Thread" gets multiple values for keyword argument "target" [misc]
```
##########
File path: python/pyspark/util.py
##########
@@ -369,39 +367,49 @@ class InheritableThread(threading.Thread):
This API is experimental.
"""
- def __init__(self, target, *args, **kwargs):
+ def __init__(self, target: Callable, *args: Any, **kwargs: Any):
from pyspark import SparkContext
- if isinstance(SparkContext._gateway, ClientServer):
+ if isinstance(SparkContext._gateway, ClientServer): # type: ignore[attr-defined]
# Here's when the pinned-thread mode (PYSPARK_PIN_THREAD) is on.
- def copy_local_properties(*a, **k):
+ def copy_local_properties(*a: Any, **k: Any) -> Any:
# self._props is set before starting the thread to match the behavior with JVM.
assert hasattr(self, "_props")
- SparkContext._active_spark_context._jsc.sc().setLocalProperties(self._props)
+ SparkContext._active_spark_context._jsc.sc().setLocalProperties( # type: ignore[attr-defined]
+ self._props
+ )
try:
return target(*a, **k)
finally:
InheritableThread._clean_py4j_conn_for_current_thread()
- super(InheritableThread, self).__init__(target=copy_local_properties, *args, **kwargs)
+ super(InheritableThread, self).__init__(
+ target=copy_local_properties, *args, **kwargs # type: ignore[misc]
+ )
Review comment:
```
python/pyspark/util.py:386: error: "__init__" of "Thread" gets multiple values for keyword argument "target" [misc]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org