You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/10/11 07:08:57 UTC
[GitHub] [spark] dchvn opened a new pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
dchvn opened a new pull request #34238:
URL: https://github.com/apache/spark/pull/34238
### What changes were proposed in this pull request?
Inline type hints for ```SparkContext```
### Why are the changes needed?
Inline type hints for ```SparkContext```
After this PR, we can remove multiple
```python
# type: ignore[attr-defined]
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
exist test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-949302633
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48999/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940639990
**[Test build #144120 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144120/testReport)** for PR 34238 at commit [`baaf509`](https://github.com/apache/spark/commit/baaf509e2d33e219cebd9956a3ada3d8a4880cd5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-941971945
**[Test build #144184 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144184/testReport)** for PR 34238 at commit [`6c14416`](https://github.com/apache/spark/commit/6c1441612ce752715176aa187aaed4fcc7539f5b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r725908729
##########
File path: python/pyspark/conf.py
##########
@@ -133,39 +141,52 @@ def set(self, key, value):
self._conf[key] = str(value)
return self
- def setIfMissing(self, key, value):
+ def setIfMissing(self, key: str, value: str) -> "SparkConf":
"""Set a configuration property, if not already set."""
if self.get(key) is None:
self.set(key, value)
return self
- def setMaster(self, value):
+ def setMaster(self, value: str) -> "SparkConf":
"""Set master URL to connect to."""
self.set("spark.master", value)
return self
- def setAppName(self, value):
+ def setAppName(self, value: str) -> "SparkConf":
"""Set application name."""
self.set("spark.app.name", value)
return self
- def setSparkHome(self, value):
+ def setSparkHome(self, value: str) -> "SparkConf":
"""Set path where Spark is installed on worker nodes."""
self.set("spark.home", value)
return self
- def setExecutorEnv(self, key=None, value=None, pairs=None):
+ @overload
+ def setExecutorEnv(self, key: str, value: str) -> "SparkConf":
+ ...
+
+ @overload
+ def setExecutorEnv(self, *, pairs: List[Tuple[str, str]]) -> "SparkConf":
+ ...
+
+ def setExecutorEnv(
+ self,
+ key: Optional[str] = None,
+ value: Optional[str] = None,
+ pairs: Optional[List[Tuple[str, str]]] = None
+ ) -> "SparkConf":
"""Set an environment variable to be passed to executors."""
if (key is not None and pairs is not None) or (key is None and pairs is None):
raise RuntimeError("Either pass one key-value pair or a list of pairs")
- elif key is not None:
+ elif key is not None and value is not None:
Review comment:
Add check ```value is not None``` in case pass one key-value pair
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-939887018
General note ‒ let's avoid changes to the actual code here on focus on type hints migration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r725990291
##########
File path: python/pyspark/conf.py
##########
@@ -195,21 +205,21 @@ def get(self, key, defaultValue=None):
else:
return self._conf.get(key, defaultValue)
- def getAll(self):
+ def getAll(self) -> List[Tuple[str, str]]:
"""Get all values as a list of key-value pairs."""
if self._jconf is not None:
return [(elem._1(), elem._2()) for elem in self._jconf.getAll()]
else:
- return self._conf.items()
+ return [(k, v) for k, v in self._conf.items()]
Review comment:
Maybe let's modify signature
https://github.com/apache/spark/blob/5e96cb11df8fbb9be320e9398d02f33bbd57b192/python/pyspark/conf.py#L208
to return `Collection[Tuple[str, str]]` for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-945282806
**[Test build #144337 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144337/testReport)** for PR 34238 at commit [`2c2e120`](https://github.com/apache/spark/commit/2c2e1204c7d01faa801f45b62a5cba88475f6ac9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940693874
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48597/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-941986663
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48663/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940285723
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48572/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-939969580
**[Test build #144082 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144082/testReport)** for PR 34238 at commit [`1af4cde`](https://github.com/apache/spark/commit/1af4cdee7180f3bbf730964c101e811bdeab3511).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r726401488
##########
File path: python/pyspark/context.py
##########
@@ -540,13 +614,13 @@ def parallelize(self, c, numSlices=None):
size = len(c)
if size == 0:
return self.parallelize([], numSlices)
- step = c[1] - c[0] if size > 1 else 1
- start0 = c[0]
+ step = c[1] - c[0] if size > 1 else 1 # type: ignore[index]
Review comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-939879665
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144073/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
dchvn commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-939867520
CC @zero323 @ueshin @HyukjinKwon FYI, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940265531
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144094/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-941992828
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-939969580
**[Test build #144082 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144082/testReport)** for PR 34238 at commit [`1af4cde`](https://github.com/apache/spark/commit/1af4cdee7180f3bbf730964c101e811bdeab3511).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r726038993
##########
File path: python/pyspark/files.py
##########
@@ -30,23 +35,23 @@ class SparkFiles(object):
instances.
"""
- _root_directory = None
+ _root_directory = None # type: str
_is_running_on_worker = False
- _sc = None
+ _sc = None # type: SparkContext
Review comment:
is it like
```python
_root_directory: Optional[str] = None
_is_running_on_worker: bool = False
_sc: Optional["SparkContext"] = None
```
thanks,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-941956000
**[Test build #144184 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144184/testReport)** for PR 34238 at commit [`6c14416`](https://github.com/apache/spark/commit/6c1441612ce752715176aa187aaed4fcc7539f5b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-945312258
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144337/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r725992924
##########
File path: python/pyspark/files.py
##########
@@ -30,23 +35,23 @@ class SparkFiles(object):
instances.
"""
- _root_directory = None
+ _root_directory = None # type: str
_is_running_on_worker = False
- _sc = None
+ _sc = None # type: SparkContext
Review comment:
Should we use PEP 526 syntax for these?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940654515
**[Test build #144120 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144120/testReport)** for PR 34238 at commit [`baaf509`](https://github.com/apache/spark/commit/baaf509e2d33e219cebd9956a3ada3d8a4880cd5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-944901042
**[Test build #144325 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144325/testReport)** for PR 34238 at commit [`6442db6`](https://github.com/apache/spark/commit/6442db6e4c13ba8dce7c8f6225b5cd0e4286de98).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-945273511
**[Test build #144337 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144337/testReport)** for PR 34238 at commit [`2c2e120`](https://github.com/apache/spark/commit/2c2e1204c7d01faa801f45b62a5cba88475f6ac9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-954018972
> This PR seem not good enough, I will close it and resolve `conf.py` and `file.py` in separated PRs.
@dchvn Please keep in mind that reviewers volunteer their time here :) It is a large project, with relatively strict policies and unless your PR is outright rejected or closed during stale PR sweep, temporary inactivity doesn't really mean anything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-949284374
**[Test build #144528 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144528/testReport)** for PR 34238 at commit [`45e0406`](https://github.com/apache/spark/commit/45e0406c842f9787c2a376a2431677865301817c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r726403990
##########
File path: python/pyspark/conf.py
##########
@@ -118,13 +126,13 @@ def __init__(self, loadDefaults=True, _jvm=None, _jconf=None):
if _jvm is not None:
# JVM is created, so create self._jconf directly through JVM
self._jconf = _jvm.SparkConf(loadDefaults)
- self._conf = None
+ self._conf: Dict[str, str] = None # type: ignore[assignment]
Review comment:
Thank you, updated. Could you take another look?
##########
File path: python/pyspark/context.py
##########
@@ -118,20 +134,36 @@ class SparkContext(object):
ValueError: ...
"""
- _gateway = None
- _jvm = None
+ _gateway = None # type: JavaGateway
+ _jvm = None # type: JavaObject
_next_accum_id = 0
- _active_spark_context = None
+ _active_spark_context = None # type: SparkContext
Review comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-939748190
**[Test build #144070 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144070/testReport)** for PR 34238 at commit [`5e96cb1`](https://github.com/apache/spark/commit/5e96cb11df8fbb9be320e9398d02f33bbd57b192).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-942022998
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48663/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-941956000
**[Test build #144184 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144184/testReport)** for PR 34238 at commit [`6c14416`](https://github.com/apache/spark/commit/6c1441612ce752715176aa187aaed4fcc7539f5b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r725979517
##########
File path: python/pyspark/conf.py
##########
@@ -133,39 +141,52 @@ def set(self, key, value):
self._conf[key] = str(value)
return self
- def setIfMissing(self, key, value):
+ def setIfMissing(self, key: str, value: str) -> "SparkConf":
"""Set a configuration property, if not already set."""
if self.get(key) is None:
self.set(key, value)
return self
- def setMaster(self, value):
+ def setMaster(self, value: str) -> "SparkConf":
"""Set master URL to connect to."""
self.set("spark.master", value)
return self
- def setAppName(self, value):
+ def setAppName(self, value: str) -> "SparkConf":
"""Set application name."""
self.set("spark.app.name", value)
return self
- def setSparkHome(self, value):
+ def setSparkHome(self, value: str) -> "SparkConf":
"""Set path where Spark is installed on worker nodes."""
self.set("spark.home", value)
return self
- def setExecutorEnv(self, key=None, value=None, pairs=None):
+ @overload
+ def setExecutorEnv(self, key: str, value: str) -> "SparkConf":
+ ...
+
+ @overload
+ def setExecutorEnv(self, *, pairs: List[Tuple[str, str]]) -> "SparkConf":
+ ...
+
+ def setExecutorEnv(
+ self,
+ key: Optional[str] = None,
+ value: Optional[str] = None,
+ pairs: Optional[List[Tuple[str, str]]] = None
+ ) -> "SparkConf":
"""Set an environment variable to be passed to executors."""
if (key is not None and pairs is not None) or (key is None and pairs is None):
raise RuntimeError("Either pass one key-value pair or a list of pairs")
- elif key is not None:
+ elif key is not None and value is not None:
Review comment:
`None` is an accepted value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r726047935
##########
File path: python/pyspark/context.py
##########
@@ -285,24 +340,27 @@ def _do_init(self, master, appName, sparkHome, pyFiles, environment, batchSize,
dump_path = self._conf.get("spark.python.profile.dump", None)
self.profiler_collector = ProfilerCollector(profiler_cls, dump_path)
else:
- self.profiler_collector = None
+ self.profiler_collector = None # type: ignore[assignment]
# create a signal handler which would be invoked on receiving SIGINT
- def signal_handler(signal, frame):
+ def signal_handler(signal: Any, frame: Any) -> None:
Review comment:
thank you, updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-939861020
**[Test build #144073 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144073/testReport)** for PR 34238 at commit [`5f3812a`](https://github.com/apache/spark/commit/5f3812ae8679e40e7e3f77cb4ad90065789026d0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940000142
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48560/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-939763302
**[Test build #144070 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144070/testReport)** for PR 34238 at commit [`5e96cb1`](https://github.com/apache/spark/commit/5e96cb11df8fbb9be320e9398d02f33bbd57b192).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940373351
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48572/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-941956000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940682849
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48597/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-944907775
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-949284374
**[Test build #144528 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144528/testReport)** for PR 34238 at commit [`45e0406`](https://github.com/apache/spark/commit/45e0406c842f9787c2a376a2431677865301817c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-945332903
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48816/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-949294399
**[Test build #144528 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144528/testReport)** for PR 34238 at commit [`45e0406`](https://github.com/apache/spark/commit/45e0406c842f9787c2a376a2431677865301817c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-939872536
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48551/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-939782079
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48548/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-939879665
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144073/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-941956000
**[Test build #144184 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144184/testReport)** for PR 34238 at commit [`6c14416`](https://github.com/apache/spark/commit/6c1441612ce752715176aa187aaed4fcc7539f5b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r726415692
##########
File path: python/pyspark/sql/pandas/conversion.py
##########
@@ -551,7 +551,7 @@ def create_RDD_server():
._serialize_to_jvm(arrow_data, ser, reader_func, create_RDD_server)
)
jdf = (
- self._jvm # type: ignore[attr-defined]
Review comment:
I will make a follow up PR to remove ```# type: ignore[attr-defined]``` for attributes of ```SparkContext```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r728011292
##########
File path: python/pyspark/context.py
##########
@@ -118,20 +125,36 @@ class SparkContext(object):
ValueError: ...
"""
- _gateway = None
- _jvm = None
- _next_accum_id = 0
- _active_spark_context = None
+ _gateway: Optional[JavaGateway] = None
+ _jvm: Optional[JavaObject] = None
+ _next_accum_id: int = 0
+ _active_spark_context: Optional["SparkContext"] = None
Review comment:
Thanks for clarification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-939826648
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48548/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-939923499
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48551/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r726038274
##########
File path: python/pyspark/conf.py
##########
@@ -118,13 +126,13 @@ def __init__(self, loadDefaults=True, _jvm=None, _jconf=None):
if _jvm is not None:
# JVM is created, so create self._jconf directly through JVM
self._jconf = _jvm.SparkConf(loadDefaults)
- self._conf = None
+ self._conf: Dict[str, str] = None # type: ignore[assignment]
Review comment:
Should we add check ```_conf is not None``` in case https://github.com/apache/spark/blob/5f3812ae8679e40e7e3f77cb4ad90065789026d0/python/pyspark/conf.py#L141
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r726047716
##########
File path: python/pyspark/conf.py
##########
@@ -133,39 +141,52 @@ def set(self, key, value):
self._conf[key] = str(value)
return self
- def setIfMissing(self, key, value):
+ def setIfMissing(self, key: str, value: str) -> "SparkConf":
"""Set a configuration property, if not already set."""
if self.get(key) is None:
self.set(key, value)
return self
- def setMaster(self, value):
+ def setMaster(self, value: str) -> "SparkConf":
"""Set master URL to connect to."""
self.set("spark.master", value)
return self
- def setAppName(self, value):
+ def setAppName(self, value: str) -> "SparkConf":
"""Set application name."""
self.set("spark.app.name", value)
return self
- def setSparkHome(self, value):
+ def setSparkHome(self, value: str) -> "SparkConf":
"""Set path where Spark is installed on worker nodes."""
self.set("spark.home", value)
return self
- def setExecutorEnv(self, key=None, value=None, pairs=None):
+ @overload
+ def setExecutorEnv(self, key: str, value: str) -> "SparkConf":
+ ...
+
+ @overload
+ def setExecutorEnv(self, *, pairs: List[Tuple[str, str]]) -> "SparkConf":
+ ...
+
+ def setExecutorEnv(
+ self,
+ key: Optional[str] = None,
+ value: Optional[str] = None,
+ pairs: Optional[List[Tuple[str, str]]] = None
+ ) -> "SparkConf":
"""Set an environment variable to be passed to executors."""
if (key is not None and pairs is not None) or (key is None and pairs is None):
raise RuntimeError("Either pass one key-value pair or a list of pairs")
- elif key is not None:
+ elif key is not None and value is not None:
Review comment:
thanks, updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
dchvn commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-954332292
Yes, I understand. I just separate files to review easier. Thanks for your help all!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r734034297
##########
File path: python/pyspark/context.py
##########
@@ -255,7 +298,8 @@ def _do_init(self, master, appName, sparkHome, pyFiles, environment, batchSize,
# Deploy code dependencies set by spark-submit; these will already have been added
# with SparkContext.addFile, so we just need to add them to the PYTHONPATH
- for path in self._conf.get("spark.submit.pyFiles", "").split(","):
+ for path in self._conf.get(
+ "spark.submit.pyFiles", "").split(","): # type: ignore[union-attr]
Review comment:
Do we need the ignore here?
##########
File path: python/pyspark/context.py
##########
@@ -206,26 +239,36 @@ def _do_init(self, master, appName, sparkHome, pyFiles, environment, batchSize,
self.environment["PYTHONHASHSEED"] = os.environ.get("PYTHONHASHSEED", "0")
# Create the Java SparkContext through Py4J
- self._jsc = jsc or self._initialize_context(self._conf._jconf)
+ self._jsc: JavaObject = jsc or self._initialize_context(self._conf._jconf)
# Reset the SparkConf to the one actually used by the SparkContext in JVM.
- self._conf = SparkConf(_jconf=self._jsc.sc().conf())
+ self._conf = SparkConf(_jconf=self._jsc.sc().conf()) # type: ignore[union-attr]
Review comment:
Do we need the ignore here?
##########
File path: python/pyspark/files.py
##########
@@ -30,29 +35,30 @@ class SparkFiles(object):
instances.
"""
- _root_directory = None
- _is_running_on_worker = False
- _sc = None
+ _root_directory: Optional[str] = None
+ _is_running_on_worker: bool = False
+ _sc: Optional["SparkContext"] = None
Review comment:
Those are `ClassVar`?
##########
File path: python/pyspark/context.py
##########
@@ -565,16 +633,24 @@ def f(split, iterator):
batchSize = max(1, min(len(c) // numSlices, self._batchSize or 1024))
serializer = BatchedSerializer(self._unbatched_serializer, batchSize)
- def reader_func(temp_filename):
- return self._jvm.PythonRDD.readRDDFromFile(self._jsc, temp_filename, numSlices)
+ def reader_func(temp_filename: str) -> JavaObject:
+ return self._jvm.PythonRDD.readRDDFromFile( # type: ignore[union-attr]
+ self._jsc, temp_filename, numSlices)
- def createRDDServer():
- return self._jvm.PythonParallelizeServer(self._jsc.sc(), numSlices)
+ def createRDDServer() -> JavaObject:
+ return self._jvm.PythonParallelizeServer( # type: ignore[union-attr]
+ self._jsc.sc(), numSlices)
jrdd = self._serialize_to_jvm(c, serializer, reader_func, createRDDServer)
return RDD(jrdd, self, serializer)
- def _serialize_to_jvm(self, data, serializer, reader_func, createRDDServer):
+ def _serialize_to_jvm(
+ self,
+ data: object,
Review comment:
`Any`?
##########
File path: python/pyspark/context.py
##########
@@ -959,37 +1084,42 @@ def union(self, rdds):
>>> sorted(sc.union([textFile, parallelized]).collect())
['Hello', 'World!']
"""
- first_jrdd_deserializer = rdds[0]._jrdd_deserializer
- if any(x._jrdd_deserializer != first_jrdd_deserializer for x in rdds):
- rdds = [x._reserialize() for x in rdds]
+ first_jrdd_deserializer = rdds[0]._jrdd_deserializer # type: ignore[index]
+ if any(
+ x._jrdd_deserializer != first_jrdd_deserializer # type: ignore[attr-defined]
+ for x in rdds
+ ):
+ rdds = [x._reserialize() for x in rdds] # type: ignore[attr-defined]
gw = SparkContext._gateway
jvm = SparkContext._jvm
- jrdd_cls = jvm.org.apache.spark.api.java.JavaRDD
- jpair_rdd_cls = jvm.org.apache.spark.api.java.JavaPairRDD
- jdouble_rdd_cls = jvm.org.apache.spark.api.java.JavaDoubleRDD
- if is_instance_of(gw, rdds[0]._jrdd, jrdd_cls):
+ jrdd_cls = jvm.org.apache.spark.api.java.JavaRDD # type: ignore[union-attr]
+ jpair_rdd_cls = jvm.org.apache.spark.api.java.JavaPairRDD # type: ignore[union-attr]
+ jdouble_rdd_cls = jvm.org.apache.spark.api.java.JavaDoubleRDD # type: ignore[union-attr]
+ if is_instance_of(gw, rdds[0]._jrdd, jrdd_cls): # type: ignore[index]
cls = jrdd_cls
- elif is_instance_of(gw, rdds[0]._jrdd, jpair_rdd_cls):
+ elif is_instance_of(gw, rdds[0]._jrdd, jpair_rdd_cls): # type: ignore[index]
cls = jpair_rdd_cls
- elif is_instance_of(gw, rdds[0]._jrdd, jdouble_rdd_cls):
+ elif is_instance_of(gw, rdds[0]._jrdd, jdouble_rdd_cls): # type: ignore[index]
cls = jdouble_rdd_cls
else:
- cls_name = rdds[0]._jrdd.getClass().getCanonicalName()
+ cls_name = rdds[0]._jrdd.getClass().getCanonicalName() # type: ignore[index]
raise TypeError("Unsupported Java RDD class %s" % cls_name)
- jrdds = gw.new_array(cls, len(rdds))
- for i in range(0, len(rdds)):
- jrdds[i] = rdds[i]._jrdd
- return RDD(self._jsc.union(jrdds), self, rdds[0]._jrdd_deserializer)
+ jrdds = gw.new_array(cls, len(rdds)) # type: ignore[arg-type, union-attr]
+ for i in range(0, len(rdds)): # type: ignore[arg-type]
+ jrdds[i] = rdds[i]._jrdd # type: ignore[index]
+ return RDD(self._jsc.union(jrdds), self, rdds[0]._jrdd_deserializer) # type: ignore[index]
- def broadcast(self, value):
+ def broadcast(self, value: T) -> "Broadcast[T]":
"""
Broadcast a read-only variable to the cluster, returning a :class:`Broadcast`
object for reading it in distributed functions. The variable will
be sent to each cluster only once.
"""
return Broadcast(self, value, self._pickled_broadcast_vars)
- def accumulator(self, value, accum_param=None):
+ def accumulator(
+ self, value: T, accum_param: "Optional[AccumulatorParam[T]]" = None
Review comment:
`Optional["AccumulatorParam[T]"]`
##########
File path: python/pyspark/context.py
##########
@@ -118,20 +125,36 @@ class SparkContext(object):
ValueError: ...
"""
- _gateway = None
- _jvm = None
- _next_accum_id = 0
- _active_spark_context = None
+ _gateway: Optional[JavaGateway] = None
+ _jvm: Optional[JavaObject] = None
+ _next_accum_id: int = 0
+ _active_spark_context: Optional["SparkContext"] = None
_lock = RLock()
Review comment:
Those should be `ClassVar`?
##########
File path: python/pyspark/context.py
##########
@@ -930,16 +1055,16 @@ def hadoopRDD(self, inputFormatClass, keyClass, valueClass, keyConverter=None,
Java object. (default 0, choose batchSize automatically)
"""
jconf = self._dictToJavaMap(conf)
- jrdd = self._jvm.PythonRDD.hadoopRDD(self._jsc, inputFormatClass, keyClass,
- valueClass, keyConverter, valueConverter,
- jconf, batchSize)
+ jrdd = self._jvm.PythonRDD.hadoopRDD( # type: ignore[union-attr]
+ self._jsc, inputFormatClass, keyClass, valueClass, keyConverter, valueConverter, jconf,
+ batchSize)
return RDD(jrdd, self)
- def _checkpointFile(self, name, input_deserializer):
+ def _checkpointFile(self, name: str, input_deserializer: Serializer) -> RDD:
jrdd = self._jsc.checkpointFile(name)
return RDD(jrdd, self, input_deserializer)
- def union(self, rdds):
+ def union(self, rdds: "Iterable[RDD[T]]") -> "RDD[T]":
Review comment:
`Iterable["RDD[T]"]`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-949323348
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48999/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-949333656
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48999/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r726220726
##########
File path: python/pyspark/context.py
##########
@@ -118,20 +134,36 @@ class SparkContext(object):
ValueError: ...
"""
- _gateway = None
- _jvm = None
+ _gateway = None # type: JavaGateway
+ _jvm = None # type: JavaObject
_next_accum_id = 0
- _active_spark_context = None
+ _active_spark_context = None # type: SparkContext
_lock = RLock()
- _python_includes = None # zip and egg files that need to be added to PYTHONPATH
-
- PACKAGE_EXTENSIONS = ('.zip', '.egg', '.jar')
-
- def __init__(self, master=None, appName=None, sparkHome=None, pyFiles=None,
- environment=None, batchSize=0, serializer=PickleSerializer(), conf=None,
- gateway=None, jsc=None, profiler_cls=BasicProfiler):
- if (conf is None or
- conf.get("spark.executor.allowSparkContext", "false").lower() != "true"):
+ # zip and egg files that need to be added to PYTHONPATH
+ _python_includes: List[str] = None # type: ignore[assignment]
+
+ PACKAGE_EXTENSIONS: Iterable[str] = ('.zip', '.egg', '.jar')
+
+ def __init__(
+ self,
+ master: Optional[str] = None,
+ appName: Optional[str] = None,
+ sparkHome: Optional[str] = None,
+ pyFiles: Optional[List[str]] = None,
+ environment: Optional[Dict[str, str]] = None,
+ batchSize: int = 0,
+ serializer: Serializer = PickleSerializer(),
+ conf: Optional[SparkConf] = None,
+ gateway: Optional[JavaGateway] = None,
+ jsc: Optional[JavaObject] = None,
+ profiler_cls: type = BasicProfiler,
+ ) -> None:
+ if (
+ conf is None
+ or conf.get(
+ "spark.executor.allowSparkContext", "false"
+ ).lower() != "true" # type: ignore[union-attr]
Review comment:
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940373385
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48572/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940666066
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144120/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940596726
**[Test build #144114 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144114/testReport)** for PR 34238 at commit [`9df286c`](https://github.com/apache/spark/commit/9df286c5fb9958d22771b063e0feeaf2d9bf0ffb).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940616187
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48591/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940661641
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48597/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r726048340
##########
File path: python/pyspark/conf.py
##########
@@ -195,21 +205,21 @@ def get(self, key, defaultValue=None):
else:
return self._conf.get(key, defaultValue)
- def getAll(self):
+ def getAll(self) -> List[Tuple[str, str]]:
"""Get all values as a list of key-value pairs."""
if self._jconf is not None:
return [(elem._1(), elem._2()) for elem in self._jconf.getAll()]
else:
- return self._conf.items()
+ return [(k, v) for k, v in self._conf.items()]
Review comment:
thanks, updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r728011292
##########
File path: python/pyspark/context.py
##########
@@ -118,20 +125,36 @@ class SparkContext(object):
ValueError: ...
"""
- _gateway = None
- _jvm = None
- _next_accum_id = 0
- _active_spark_context = None
+ _gateway: Optional[JavaGateway] = None
+ _jvm: Optional[JavaObject] = None
+ _next_accum_id: int = 0
+ _active_spark_context: Optional["SparkContext"] = None
Review comment:
Thanks for clarification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r730017148
##########
File path: python/pyspark/sql/functions.py
##########
@@ -71,7 +71,7 @@ def _get_get_jvm_function(name: str, sc: SparkContext) -> Callable:
Retrieves JVM function identified by name from
Java gateway associated with sc.
"""
- return getattr(sc._jvm.functions, name) # type: ignore[attr-defined]
+ return getattr(sc._jvm.functions, name) # type: ignore[attr-defined, union-attr]
Review comment:
Thanks!
##########
File path: python/pyspark/conf.py
##########
@@ -118,43 +126,56 @@ def __init__(self, loadDefaults=True, _jvm=None, _jconf=None):
if _jvm is not None:
# JVM is created, so create self._jconf directly through JVM
self._jconf = _jvm.SparkConf(loadDefaults)
- self._conf = None
+ self._conf: Optional[Dict[str, str]] = None
else:
# JVM is not created, so store data in self._conf first
self._jconf = None
self._conf = {}
- def set(self, key, value):
+ def set(self, key: str, value: Optional[str]) -> "SparkConf":
Review comment:
Is it possible for `value` to be `Any`?
##########
File path: python/pyspark/conf.py
##########
@@ -118,43 +126,56 @@ def __init__(self, loadDefaults=True, _jvm=None, _jconf=None):
if _jvm is not None:
# JVM is created, so create self._jconf directly through JVM
self._jconf = _jvm.SparkConf(loadDefaults)
- self._conf = None
+ self._conf: Optional[Dict[str, str]] = None
else:
# JVM is not created, so store data in self._conf first
self._jconf = None
self._conf = {}
- def set(self, key, value):
+ def set(self, key: str, value: Optional[str]) -> "SparkConf":
Review comment:
Is it possible for `value` to be `Optional[Any]`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
dchvn commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-953930854
This PR seem not good enough, I will close it and resolve ```conf.py``` and ```file.py``` in separated PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-945273511
**[Test build #144337 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144337/testReport)** for PR 34238 at commit [`2c2e120`](https://github.com/apache/spark/commit/2c2e1204c7d01faa801f45b62a5cba88475f6ac9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
dchvn commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-945329460
Ping @zero323 , Could you please take another look ? thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r730279138
##########
File path: python/pyspark/conf.py
##########
@@ -118,43 +126,56 @@ def __init__(self, loadDefaults=True, _jvm=None, _jconf=None):
if _jvm is not None:
# JVM is created, so create self._jconf directly through JVM
self._jconf = _jvm.SparkConf(loadDefaults)
- self._conf = None
+ self._conf: Optional[Dict[str, str]] = None
else:
# JVM is not created, so store data in self._conf first
self._jconf = None
self._conf = {}
- def set(self, key, value):
+ def set(self, key: str, value: Optional[str]) -> "SparkConf":
Review comment:
Yes, thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940011231
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144082/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn edited a comment on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
dchvn edited a comment on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-939867520
CC @zero323 @ueshin @HyukjinKwon @xinrong-databricks FYI, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-939832601
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48548/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-939786069
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144070/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940037452
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48560/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r726051251
##########
File path: python/pyspark/context.py
##########
@@ -118,20 +134,36 @@ class SparkContext(object):
ValueError: ...
"""
- _gateway = None
- _jvm = None
+ _gateway = None # type: JavaGateway
+ _jvm = None # type: JavaObject
_next_accum_id = 0
- _active_spark_context = None
+ _active_spark_context = None # type: SparkContext
Review comment:
when using ```Optional```, we need resolve huge of code like
```SparkContext._jvm``` by ```ignore[union-attr]``` or check ```None```.
Could please give me a suggestion ? thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-939989092
**[Test build #144082 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144082/testReport)** for PR 34238 at commit [`1af4cde`](https://github.com/apache/spark/commit/1af4cdee7180f3bbf730964c101e811bdeab3511).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r725981818
##########
File path: python/pyspark/context.py
##########
@@ -540,13 +614,13 @@ def parallelize(self, c, numSlices=None):
size = len(c)
if size == 0:
return self.parallelize([], numSlices)
- step = c[1] - c[0] if size > 1 else 1
- start0 = c[0]
+ step = c[1] - c[0] if size > 1 else 1 # type: ignore[index]
Review comment:
This should be handled with either cast or type assertion, not ignore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-939748190
**[Test build #144070 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144070/testReport)** for PR 34238 at commit [`5e96cb1`](https://github.com/apache/spark/commit/5e96cb11df8fbb9be320e9398d02f33bbd57b192).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r725847097
##########
File path: python/pyspark/conf.py
##########
@@ -195,21 +205,21 @@ def get(self, key, defaultValue=None):
else:
return self._conf.get(key, defaultValue)
- def getAll(self):
+ def getAll(self) -> List[Tuple[str, str]]:
"""Get all values as a list of key-value pairs."""
if self._jconf is not None:
return [(elem._1(), elem._2()) for elem in self._jconf.getAll()]
else:
- return self._conf.items()
+ return [(k, v) for k, v in self._conf.items()]
Review comment:
convert to ```List[Tuple[str, str]]```
##########
File path: python/pyspark/context.py
##########
@@ -285,24 +340,27 @@ def _do_init(self, master, appName, sparkHome, pyFiles, environment, batchSize,
dump_path = self._conf.get("spark.python.profile.dump", None)
self.profiler_collector = ProfilerCollector(profiler_cls, dump_path)
else:
- self.profiler_collector = None
+ self.profiler_collector = None # type: ignore[assignment]
# create a signal handler which would be invoked on receiving SIGINT
- def signal_handler(signal, frame):
+ def signal_handler(signal: Any, frame: Any) -> None:
Review comment:
I am not sure for this method!
##########
File path: python/pyspark/context.py
##########
@@ -118,20 +134,36 @@ class SparkContext(object):
ValueError: ...
"""
- _gateway = None
- _jvm = None
+ _gateway = None # type: JavaGateway
+ _jvm = None # type: JavaObject
_next_accum_id = 0
- _active_spark_context = None
+ _active_spark_context = None # type: SparkContext
_lock = RLock()
- _python_includes = None # zip and egg files that need to be added to PYTHONPATH
-
- PACKAGE_EXTENSIONS = ('.zip', '.egg', '.jar')
-
- def __init__(self, master=None, appName=None, sparkHome=None, pyFiles=None,
- environment=None, batchSize=0, serializer=PickleSerializer(), conf=None,
- gateway=None, jsc=None, profiler_cls=BasicProfiler):
- if (conf is None or
- conf.get("spark.executor.allowSparkContext", "false").lower() != "true"):
+ # zip and egg files that need to be added to PYTHONPATH
+ _python_includes: List[str] = None # type: ignore[assignment]
+
+ PACKAGE_EXTENSIONS: Iterable[str] = ('.zip', '.egg', '.jar')
+
+ def __init__(
+ self,
+ master: Optional[str] = None,
+ appName: Optional[str] = None,
+ sparkHome: Optional[str] = None,
+ pyFiles: Optional[List[str]] = None,
+ environment: Optional[Dict[str, str]] = None,
+ batchSize: int = 0,
+ serializer: Serializer = PickleSerializer(),
+ conf: Optional[SparkConf] = None,
+ gateway: Optional[JavaGateway] = None,
+ jsc: Optional[JavaObject] = None,
+ profiler_cls: type = BasicProfiler,
+ ) -> None:
+ if (
+ conf is None
+ or conf.get(
+ "spark.executor.allowSparkContext", "false"
+ ).lower() != "true" # type: ignore[union-attr]
Review comment:
Should we check ```conf.get(key, ...) != None``` ?
##########
File path: python/pyspark/context.py
##########
@@ -540,13 +614,13 @@ def parallelize(self, c, numSlices=None):
size = len(c)
if size == 0:
return self.parallelize([], numSlices)
- step = c[1] - c[0] if size > 1 else 1
- start0 = c[0]
+ step = c[1] - c[0] if size > 1 else 1 # type: ignore[index]
Review comment:
```Iterable``` is not indexable
##########
File path: python/pyspark/context.py
##########
@@ -390,17 +458,17 @@ def getOrCreate(cls, conf=None):
with SparkContext._lock:
if SparkContext._active_spark_context is None:
SparkContext(conf=conf or SparkConf())
- return SparkContext._active_spark_context
+ return SparkContext._active_spark_context # type: ignore[return-value]
Review comment:
do not need ignore here
##########
File path: python/pyspark/context.py
##########
@@ -562,19 +636,28 @@ def f(split, iterator):
# Make sure we distribute data evenly if it's smaller than self.batchSize
if "__len__" not in dir(c):
c = list(c) # Make it a list so we can compute its length
- batchSize = max(1, min(len(c) // numSlices, self._batchSize or 1024))
+ batchSize = max(
+ 1,
+ min(len(c) // numSlices, self._batchSize or 1024) # type: ignore[arg-type]
Review comment:
```Iterable``` does not have ```len```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940238822
**[Test build #144094 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144094/testReport)** for PR 34238 at commit [`8663962`](https://github.com/apache/spark/commit/866396247b72b828fa7b93e46e583db4ddc09927).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-939832601
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48548/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-945312258
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144337/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940666066
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144120/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940596726
**[Test build #144114 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144114/testReport)** for PR 34238 at commit [`9df286c`](https://github.com/apache/spark/commit/9df286c5fb9958d22771b063e0feeaf2d9bf0ffb).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-942044057
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48663/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940608574
**[Test build #144114 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144114/testReport)** for PR 34238 at commit [`9df286c`](https://github.com/apache/spark/commit/9df286c5fb9958d22771b063e0feeaf2d9bf0ffb).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940637929
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48591/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r727989028
##########
File path: python/pyspark/context.py
##########
@@ -118,20 +125,36 @@ class SparkContext(object):
ValueError: ...
"""
- _gateway = None
- _jvm = None
- _next_accum_id = 0
- _active_spark_context = None
+ _gateway: Optional[JavaGateway] = None
+ _jvm: Optional[JavaObject] = None
+ _next_accum_id: int = 0
+ _active_spark_context: Optional["SparkContext"] = None
Review comment:
> Could you explain, why it escalated to all kinds of unrelated files with all the `ignore[union-attr]`?
@zero323 Thanks for reviewing, as in this https://github.com/apache/spark/pull/34238#discussion_r726051251, when we have type hint ```Optional[...]``` for these attributes of ```SparkContext```, many places in unrelated files access them without check ```None``` for ```SparkContext``` variable, so ```mypy check``` throws error `ignore[union-attr]`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r726108860
##########
File path: python/pyspark/conf.py
##########
@@ -118,13 +126,13 @@ def __init__(self, loadDefaults=True, _jvm=None, _jconf=None):
if _jvm is not None:
# JVM is created, so create self._jconf directly through JVM
self._jconf = _jvm.SparkConf(loadDefaults)
- self._conf = None
+ self._conf: Dict[str, str] = None # type: ignore[assignment]
Review comment:
> Should we add check `_conf is not None` in case
In general access to `_conf` is conditioned on `_jconf` not being present, and if it is not, `_conf` is set within initializer. So I'd say no.
Also, if it was something to change, it would be out of scope here. Type hint migration touches a lot of code, but by itself should be safe. Adding anything to it, unless strictly necessary, not so much :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940631853
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48591/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940373385
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48572/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940614748
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144114/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940011231
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144082/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940693874
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48597/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-939833659
**[Test build #144073 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144073/testReport)** for PR 34238 at commit [`5f3812a`](https://github.com/apache/spark/commit/5f3812ae8679e40e7e3f77cb4ad90065789026d0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-949304530
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144528/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
dchvn commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-950052545
@ueshin Thanks for your help! I updated this PR. Could you take another look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-944901042
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940062620
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48560/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-942044057
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48663/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-944907775
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-939786069
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144070/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-945307034
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48816/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-945332903
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48816/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940265531
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144094/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r725977630
##########
File path: python/pyspark/context.py
##########
@@ -562,19 +636,28 @@ def f(split, iterator):
# Make sure we distribute data evenly if it's smaller than self.batchSize
if "__len__" not in dir(c):
c = list(c) # Make it a list so we can compute its length
- batchSize = max(1, min(len(c) // numSlices, self._batchSize or 1024))
+ batchSize = max(
+ 1,
+ min(len(c) // numSlices, self._batchSize or 1024) # type: ignore[arg-type]
Review comment:
This should be handled with either `cast` or type assertion, not ignore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-941992828
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144184/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940639990
**[Test build #144120 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144120/testReport)** for PR 34238 at commit [`baaf509`](https://github.com/apache/spark/commit/baaf509e2d33e219cebd9956a3ada3d8a4880cd5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r726042916
##########
File path: python/pyspark/conf.py
##########
@@ -118,13 +126,13 @@ def __init__(self, loadDefaults=True, _jvm=None, _jconf=None):
if _jvm is not None:
# JVM is created, so create self._jconf directly through JVM
self._jconf = _jvm.SparkConf(loadDefaults)
- self._conf = None
+ self._conf: Dict[str, str] = None # type: ignore[assignment]
Review comment:
about 7 case like this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940218878
**[Test build #144094 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144094/testReport)** for PR 34238 at commit [`8663962`](https://github.com/apache/spark/commit/866396247b72b828fa7b93e46e583db4ddc09927).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r725996203
##########
File path: python/pyspark/context.py
##########
@@ -118,20 +134,36 @@ class SparkContext(object):
ValueError: ...
"""
- _gateway = None
- _jvm = None
+ _gateway = None # type: JavaGateway
+ _jvm = None # type: JavaObject
_next_accum_id = 0
- _active_spark_context = None
+ _active_spark_context = None # type: SparkContext
_lock = RLock()
- _python_includes = None # zip and egg files that need to be added to PYTHONPATH
-
- PACKAGE_EXTENSIONS = ('.zip', '.egg', '.jar')
-
- def __init__(self, master=None, appName=None, sparkHome=None, pyFiles=None,
- environment=None, batchSize=0, serializer=PickleSerializer(), conf=None,
- gateway=None, jsc=None, profiler_cls=BasicProfiler):
- if (conf is None or
- conf.get("spark.executor.allowSparkContext", "false").lower() != "true"):
+ # zip and egg files that need to be added to PYTHONPATH
+ _python_includes: List[str] = None # type: ignore[assignment]
+
+ PACKAGE_EXTENSIONS: Iterable[str] = ('.zip', '.egg', '.jar')
Review comment:
`Tuple[str, ...]`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-939906890
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48551/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-939923499
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48551/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-939833659
**[Test build #144073 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144073/testReport)** for PR 34238 at commit [`5f3812a`](https://github.com/apache/spark/commit/5f3812ae8679e40e7e3f77cb4ad90065789026d0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r726003319
##########
File path: python/pyspark/conf.py
##########
@@ -18,6 +18,9 @@
__all__ = ['SparkConf']
import sys
+from typing import Optional, List, Tuple, Dict, overload
+
+from py4j.java_gateway import JVMView, JavaObject
Review comment:
NM. That's probably overthinking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-945319210
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48816/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-949304530
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144528/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn closed pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
dchvn closed pull request #34238:
URL: https://github.com/apache/spark/pull/34238
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-949333656
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48999/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r726405407
##########
File path: python/pyspark/context.py
##########
@@ -562,19 +636,28 @@ def f(split, iterator):
# Make sure we distribute data evenly if it's smaller than self.batchSize
if "__len__" not in dir(c):
c = list(c) # Make it a list so we can compute its length
- batchSize = max(1, min(len(c) // numSlices, self._batchSize or 1024))
+ batchSize = max(
+ 1,
+ min(len(c) // numSlices, self._batchSize or 1024) # type: ignore[arg-type]
Review comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940614748
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144114/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r726435163
##########
File path: python/pyspark/sql/functions.py
##########
@@ -71,7 +71,7 @@ def _get_get_jvm_function(name: str, sc: SparkContext) -> Callable:
Retrieves JVM function identified by name from
Java gateway associated with sc.
"""
- return getattr(sc._jvm.functions, name) # type: ignore[attr-defined]
+ return getattr(sc._jvm.functions, name) # type: ignore[attr-defined, union-attr]
Review comment:
I will make a Follow-up PR to remove ```# type: ignore[attr-defined]``` for attributes access of ```SparkContext```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r725988872
##########
File path: python/pyspark/context.py
##########
@@ -285,24 +340,27 @@ def _do_init(self, master, appName, sparkHome, pyFiles, environment, batchSize,
dump_path = self._conf.get("spark.python.profile.dump", None)
self.profiler_collector = ProfilerCollector(profiler_cls, dump_path)
else:
- self.profiler_collector = None
+ self.profiler_collector = None # type: ignore[assignment]
# create a signal handler which would be invoked on receiving SIGINT
- def signal_handler(signal, frame):
+ def signal_handler(signal: Any, frame: Any) -> None:
Review comment:
`signal.signal` [is called with `_HANDLER`](https://github.com/python/typeshed/blob/ee487304d76c671aa353a15e34bc1102bffa2362/stdlib/signal.pyi#L188)
```python
def signal(__signalnum: _SIGNUM, __handler: _HANDLER) -> _HANDLER: ...
```
which [is an alias](https://github.com/python/typeshed/blob/ee487304d76c671aa353a15e34bc1102bffa2362/stdlib/signal.pyi#L83) for:
```python
_HANDLER = Union[Callable[[int, Optional[FrameType]], Any], int, Handlers, None]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940637929
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48591/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940218878
**[Test build #144094 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144094/testReport)** for PR 34238 at commit [`8663962`](https://github.com/apache/spark/commit/866396247b72b828fa7b93e46e583db4ddc09927).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r725995807
##########
File path: python/pyspark/context.py
##########
@@ -118,20 +134,36 @@ class SparkContext(object):
ValueError: ...
"""
- _gateway = None
- _jvm = None
+ _gateway = None # type: JavaGateway
+ _jvm = None # type: JavaObject
_next_accum_id = 0
- _active_spark_context = None
+ _active_spark_context = None # type: SparkContext
Review comment:
Seems like these should be `Optional`,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r725994872
##########
File path: python/pyspark/conf.py
##########
@@ -118,13 +126,13 @@ def __init__(self, loadDefaults=True, _jvm=None, _jconf=None):
if _jvm is not None:
# JVM is created, so create self._jconf directly through JVM
self._jconf = _jvm.SparkConf(loadDefaults)
- self._conf = None
+ self._conf: Dict[str, str] = None # type: ignore[assignment]
Review comment:
Any reason not have `Optional[Dict[str, str]]`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-940062620
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48560/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r725992060
##########
File path: python/pyspark/conf.py
##########
@@ -18,6 +18,9 @@
__all__ = ['SparkConf']
import sys
+from typing import Optional, List, Tuple, Dict, overload
+
+from py4j.java_gateway import JVMView, JavaObject
Review comment:
It might be better to keep it in `TYPE_CHECKING` conditional, since this is what we do in other migrated contexts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34238:
URL: https://github.com/apache/spark/pull/34238#discussion_r726038993
##########
File path: python/pyspark/files.py
##########
@@ -30,23 +35,23 @@ class SparkFiles(object):
instances.
"""
- _root_directory = None
+ _root_directory = None # type: str
_is_running_on_worker = False
- _sc = None
+ _sc = None # type: SparkContext
Review comment:
does it like
```python
_root_directory: Optional[str] = None
_is_running_on_worker: bool = False
_sc: Optional["SparkContext"] = None
```
thanks,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-941992828
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34238: [SPARK-36969][PYTHON] Inline type hints for SparkContext
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34238:
URL: https://github.com/apache/spark/pull/34238#issuecomment-941992828
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144184/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org