You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/04/09 15:29:38 UTC
[GitHub] [spark] Yikun opened a new pull request #32110: [SPARK-34630][PYTHON] Add typehint for pyspark.__version__
Yikun opened a new pull request #32110:
URL: https://github.com/apache/spark/pull/32110
### What changes were proposed in this pull request?
This PR adds the typehint of pyspark.__version__, which was mentioned in [SPARK-34630](https://issues.apache.org/jira/browse/SPARK-34630).
### Why are the changes needed?
There were some short discussion happened in https://github.com/apache/spark/pull/31823#discussion_r593830911 .
After further deep investigation on [1][2], we can see the `pyspark.__version__` is added by [setup.py](https://github.com/apache/spark/blob/c06758834e6192b1888b4a885c612a151588b390/python/setup.py#L201), it makes `__version__` embedded into pyspark module, that means the `__init__.pyi` is the right place to add the typehint for `__version__`.
So, this patch add the type hint __version__ in pyspark/__init__.pyi.
[1] [PEP-396 Module Version Numbers](https://www.python.org/dev/peps/pep-0396/)
[2] https://packaging.python.org/guides/single-sourcing-package-version/
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
1. disable the ignore_error on
https://github.com/apache/spark/blob/ee7bf7d9628384548d09ac20e30992dd705c20f4/python/mypy.ini#L132
2. Run mypy:
- Before fix
```shell
(venv) ➜ spark git:(SPARK-34629) ✗ mypy --config-file python/mypy.ini python/pyspark | grep version
python/pyspark/pandas/spark/accessors.py:884: error: Module has no attribute "__version__"
```
- After fix
```shell
(venv) ➜ spark git:(SPARK-34629) ✗ mypy --config-file python/mypy.ini python/pyspark | grep version
```
no output
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32110: [SPARK-34630][PYTHON] Add typehint for pyspark.__version__
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32110:
URL: https://github.com/apache/spark/pull/32110#issuecomment-816767378
**[Test build #137144 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137144/testReport)** for PR 32110 at commit [`2ec4f8d`](https://github.com/apache/spark/commit/2ec4f8d717e561e944bb145c4d0de0cb211a21a7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on pull request #32110: [SPARK-34630][PYTHON] Add typehint for pyspark.__version__
Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #32110:
URL: https://github.com/apache/spark/pull/32110#issuecomment-817331896
I think it will require some changes. First of all `__version__` is defined in `python.version`
https://github.com/apache/spark/blob/0e761c7307e780bd1d6cfe40e3d17ad003768411/python/pyspark/version.pyi#L19
so we shouldn't really annotate it directly in `pyspark.__init__`.
It is imported there, but not placed in `__all__`, so it formally not considered as exported.
We could cheat a bit and simply have
```python
from pyspark.version import __version__ as __version__
```
but to be consistent, we should also add `__version__` to `__all__` and drop `noqa: F401` from
https://github.com/apache/spark/blob/512f6258071666c3da07f6e6c6299d653a1cc370/python/pyspark/__init__.py#L63
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32110: [SPARK-34630][PYTHON] Add typehint for pyspark.__version__
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32110:
URL: https://github.com/apache/spark/pull/32110#issuecomment-816793344
**[Test build #137144 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137144/testReport)** for PR 32110 at commit [`2ec4f8d`](https://github.com/apache/spark/commit/2ec4f8d717e561e944bb145c4d0de0cb211a21a7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32110: [SPARK-34630][PYTHON] Add typehint for pyspark.__version__
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32110:
URL: https://github.com/apache/spark/pull/32110#issuecomment-816795749
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137144/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #32110: [SPARK-34630][PYTHON] Add typehint for pyspark.__version__
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #32110:
URL: https://github.com/apache/spark/pull/32110#issuecomment-817432476
@zero323 It makes sense to me, we should make the `__version__` as exported, and I added a followup https://github.com/apache/spark/pull/32125 to fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32110: [SPARK-34630][PYTHON] Add typehint for pyspark.__version__
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32110:
URL: https://github.com/apache/spark/pull/32110#issuecomment-816804056
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41723/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #32110: [SPARK-34630][PYTHON] Add typehint for pyspark.__version__
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #32110:
URL: https://github.com/apache/spark/pull/32110#issuecomment-816765690
cc @dannymeijer @zero323 @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32110: [SPARK-34630][PYTHON] Add typehint for pyspark.__version__
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32110:
URL: https://github.com/apache/spark/pull/32110#issuecomment-816804080
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41723/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #32110: [SPARK-34630][PYTHON] Add typehint for pyspark.__version__
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #32110:
URL: https://github.com/apache/spark/pull/32110
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32110: [SPARK-34630][PYTHON] Add typehint for pyspark.__version__
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32110:
URL: https://github.com/apache/spark/pull/32110#issuecomment-816795749
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137144/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32110: [SPARK-34630][PYTHON] Add typehint for pyspark.__version__
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32110:
URL: https://github.com/apache/spark/pull/32110#issuecomment-816804080
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41723/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32110: [SPARK-34630][PYTHON] Add typehint for pyspark.__version__
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32110:
URL: https://github.com/apache/spark/pull/32110#issuecomment-816767378
**[Test build #137144 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137144/testReport)** for PR 32110 at commit [`2ec4f8d`](https://github.com/apache/spark/commit/2ec4f8d717e561e944bb145c4d0de0cb211a21a7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32110: [SPARK-34630][PYTHON] Add typehint for pyspark.__version__
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32110:
URL: https://github.com/apache/spark/pull/32110#issuecomment-817232419
Merged to master and branch-3.1.
But @zero323 feel free to post-hoc review since you're the expert here :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org