You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/10/29 07:12:56 UTC
[GitHub] [spark] dchvn opened a new pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
dchvn opened a new pull request #34435:
URL: https://github.com/apache/spark/pull/34435
### What changes were proposed in this pull request?
Inline type hints for python/pyspark/statcounter.py
### Why are the changes needed?
We can take advantage of static type checking within the functions by inlining the type hints.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Existing tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #34435:
URL: https://github.com/apache/spark/pull/34435#discussion_r750668911
##########
File path: python/pyspark/statcounter.py
##########
@@ -53,12 +54,14 @@ def merge(self, value):
return self
# Merge another StatCounter into this one, adding up the internal statistics.
- def mergeStats(self, other):
+ def mergeStats(self, other: "StatCounter") -> "StatCounter":
if not isinstance(other, StatCounter):
raise TypeError("Can only merge StatCounter but got %s" % type(other))
if other is self: # reference equality holds
- self.merge(copy.deepcopy(other)) # Avoid overwriting fields in a weird order
+ self.merge(
+ cast(float, copy.deepcopy(other))
+ ) # Avoid overwriting fields in a weird order
Review comment:
How can we cast `cast(float, copy.deepcopy(other))` here?
I guess this is a long-standing bug and should call `self.mergeStats` instead of `self.merge`? cc @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-962426321
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144953/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-971041767
**[Test build #145298 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145298/testReport)** for PR 34435 at commit [`8752dff`](https://github.com/apache/spark/commit/8752dff05af04776e3a8f0fb17c9d67fffdd8395).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-954490472
**[Test build #144742 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144742/testReport)** for PR 34435 at commit [`895e22f`](https://github.com/apache/spark/commit/895e22fd94b81cfd987624b3462e88ecb3ab0d10).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-954583732
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49211/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-961755067
**[Test build #144929 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144929/testReport)** for PR 34435 at commit [`5fdb758`](https://github.com/apache/spark/commit/5fdb75825f1cbf6635e21455bfd4fcc20a583b3d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-962433031
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49424/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-961755067
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-966063167
**[Test build #145093 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145093/testReport)** for PR 34435 at commit [`8ba331b`](https://github.com/apache/spark/commit/8ba331b1c25e1c87e65333696fc001cc7312b6e8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-971142237
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-971114405
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49768/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34435:
URL: https://github.com/apache/spark/pull/34435#discussion_r750690775
##########
File path: python/pyspark/statcounter.py
##########
@@ -53,12 +54,14 @@ def merge(self, value):
return self
# Merge another StatCounter into this one, adding up the internal statistics.
- def mergeStats(self, other):
+ def mergeStats(self, other: "StatCounter") -> "StatCounter":
if not isinstance(other, StatCounter):
raise TypeError("Can only merge StatCounter but got %s" % type(other))
if other is self: # reference equality holds
- self.merge(copy.deepcopy(other)) # Avoid overwriting fields in a weird order
+ self.merge(
+ cast(float, copy.deepcopy(other))
+ ) # Avoid overwriting fields in a weird order
Review comment:
This is nice catch @ueshin! The question is why would even get here...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34435:
URL: https://github.com/apache/spark/pull/34435#discussion_r747246702
##########
File path: python/pyspark/statcounter.py
##########
@@ -19,18 +19,19 @@
import copy
import math
+from typing import cast, Dict, Iterable, Optional, Union
try:
from numpy import maximum, minimum, sqrt
except ImportError:
- maximum = max
- minimum = min
- sqrt = math.sqrt
+ maximum = max # type: ignore
+ minimum = min # type: ignore
+ sqrt = math.sqrt # type: ignore
Review comment:
Updated. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-954490472
**[Test build #144742 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144742/testReport)** for PR 34435 at commit [`895e22f`](https://github.com/apache/spark/commit/895e22fd94b81cfd987624b3462e88ecb3ab0d10).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-972740493
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49854/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-962429523
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49424/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-961698191
In general, this should be rewritten with `Protocols` or equivalent at some point, but it is not priority and it is not in the scope of this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34435:
URL: https://github.com/apache/spark/pull/34435#discussion_r743400286
##########
File path: python/pyspark/statcounter.py
##########
@@ -19,6 +19,11 @@
import copy
import math
+from typing import cast, Any, Dict, Iterable, Optional, Union
+
+maximum: Any
+minimum: Any
+sqrt: Any
Review comment:
can we avoid this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-961830542
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49400/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34435:
URL: https://github.com/apache/spark/pull/34435#discussion_r751811590
##########
File path: python/pyspark/statcounter.py
##########
@@ -53,12 +54,12 @@ def merge(self, value):
return self
# Merge another StatCounter into this one, adding up the internal statistics.
- def mergeStats(self, other):
+ def mergeStats(self, other: "StatCounter") -> "StatCounter":
if not isinstance(other, StatCounter):
raise TypeError("Can only merge StatCounter but got %s" % type(other))
if other is self: # reference equality holds
- self.merge(copy.deepcopy(other)) # Avoid overwriting fields in a weird order
+ self.mergeStats(copy.deepcopy(other)) # Avoid overwriting fields in a weird order
Review comment:
Should I just add # type: ignore like: `self.merge(copy.deepcopy(other)) # type: ignore` and create a new Jira ticket with UT for this issue? @zero323 , @ueshin , @HyukjinKwon
Many thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34435:
URL: https://github.com/apache/spark/pull/34435#discussion_r751943438
##########
File path: python/pyspark/statcounter.py
##########
@@ -53,12 +54,12 @@ def merge(self, value):
return self
# Merge another StatCounter into this one, adding up the internal statistics.
- def mergeStats(self, other):
+ def mergeStats(self, other: "StatCounter") -> "StatCounter":
if not isinstance(other, StatCounter):
raise TypeError("Can only merge StatCounter but got %s" % type(other))
if other is self: # reference equality holds
- self.merge(copy.deepcopy(other)) # Avoid overwriting fields in a weird order
+ self.mergeStats(copy.deepcopy(other)) # Avoid overwriting fields in a weird order
Review comment:
Yeah, let's handle it in a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-954522044
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49211/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34435:
URL: https://github.com/apache/spark/pull/34435#discussion_r745226231
##########
File path: python/pyspark/statcounter.py
##########
@@ -54,12 +59,13 @@ def merge(self, value):
return self
# Merge another StatCounter into this one, adding up the internal statistics.
- def mergeStats(self, other):
+ def mergeStats(self, other: "StatCounter") -> "StatCounter":
if not isinstance(other, StatCounter):
raise TypeError("Can only merge StatCounter but got %s" % type(other))
if other is self: # reference equality holds
- self.merge(copy.deepcopy(other)) # Avoid overwriting fields in a weird order
+ self.merge(
+ copy.deepcopy(cast(float, other))) # Avoid overwriting fields in a weird order
Review comment:
updated. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-961755067
**[Test build #144929 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144929/testReport)** for PR 34435 at commit [`5fdb758`](https://github.com/apache/spark/commit/5fdb75825f1cbf6635e21455bfd4fcc20a583b3d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-961791371
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-962423466
**[Test build #144953 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144953/testReport)** for PR 34435 at commit [`65294a4`](https://github.com/apache/spark/commit/65294a4bb2db9a65fa3a0144f583b554a943fdbe).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-962433031
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49424/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-954507767
**[Test build #144742 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144742/testReport)** for PR 34435 at commit [`895e22f`](https://github.com/apache/spark/commit/895e22fd94b81cfd987624b3462e88ecb3ab0d10).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-966075656
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145093/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-966075656
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145093/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-970712981
LGTM, assuming we don't want address https://github.com/apache/spark/pull/34435#discussion_r750668911 in this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-971132481
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49768/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-972645595
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145381/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34435:
URL: https://github.com/apache/spark/pull/34435#discussion_r743400255
##########
File path: python/pyspark/statcounter.py
##########
@@ -54,12 +59,13 @@ def merge(self, value):
return self
# Merge another StatCounter into this one, adding up the internal statistics.
- def mergeStats(self, other):
+ def mergeStats(self, other: "StatCounter") -> "StatCounter":
if not isinstance(other, StatCounter):
raise TypeError("Can only merge StatCounter but got %s" % type(other))
if other is self: # reference equality holds
- self.merge(copy.deepcopy(other)) # Avoid overwriting fields in a weird order
+ self.merge(
+ copy.deepcopy(cast(float, other))) # Avoid overwriting fields in a weird order
Review comment:
this looks weird. when other is self, shouldn't `other` be `StatCounter`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-954583732
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49211/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34435:
URL: https://github.com/apache/spark/pull/34435#discussion_r745225523
##########
File path: python/pyspark/statcounter.py
##########
@@ -114,24 +120,24 @@ def variance(self):
# Return the sample variance, which corrects for bias in estimating the variance by dividing
# by N-1 instead of N.
#
- def sampleVariance(self):
+ def sampleVariance(self) -> float:
if self.n <= 1:
return float('nan')
else:
return self.m2 / (self.n - 1)
# Return the standard deviation of the values.
- def stdev(self):
+ def stdev(self) -> float:
return sqrt(self.variance())
#
# Return the sample standard deviation of the values, which corrects for bias in estimating the
# variance by dividing by N-1 instead of N.
#
- def sampleStdev(self):
+ def sampleStdev(self) -> float:
return sqrt(self.sampleVariance())
- def asDict(self, sample=False):
+ def asDict(self, sample: bool = False) -> Dict[str, Union[float, int]]:
Review comment:
updated. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34435:
URL: https://github.com/apache/spark/pull/34435#discussion_r743453323
##########
File path: python/pyspark/statcounter.py
##########
@@ -114,24 +120,24 @@ def variance(self):
# Return the sample variance, which corrects for bias in estimating the variance by dividing
# by N-1 instead of N.
#
- def sampleVariance(self):
+ def sampleVariance(self) -> float:
if self.n <= 1:
return float('nan')
else:
return self.m2 / (self.n - 1)
# Return the standard deviation of the values.
- def stdev(self):
+ def stdev(self) -> float:
return sqrt(self.variance())
#
# Return the sample standard deviation of the values, which corrects for bias in estimating the
# variance by dividing by N-1 instead of N.
#
- def sampleStdev(self):
+ def sampleStdev(self) -> float:
return sqrt(self.sampleVariance())
- def asDict(self, sample=False):
+ def asDict(self, sample: bool = False) -> Dict[str, Union[float, int]]:
Review comment:
Nit. Might be cleaner to have `Dict[str, float]`.
##########
File path: python/pyspark/statcounter.py
##########
@@ -114,24 +120,24 @@ def variance(self):
# Return the sample variance, which corrects for bias in estimating the variance by dividing
# by N-1 instead of N.
#
- def sampleVariance(self):
+ def sampleVariance(self) -> float:
if self.n <= 1:
return float('nan')
else:
return self.m2 / (self.n - 1)
# Return the standard deviation of the values.
- def stdev(self):
+ def stdev(self) -> float:
return sqrt(self.variance())
#
# Return the sample standard deviation of the values, which corrects for bias in estimating the
# variance by dividing by N-1 instead of N.
#
- def sampleStdev(self):
+ def sampleStdev(self) -> float:
return sqrt(self.sampleVariance())
- def asDict(self, sample=False):
+ def asDict(self, sample: bool = False) -> Dict[str, Union[float, int]]:
Review comment:
There are contexts, where distinguishing between these two is important (for us, it is primarily for arguments which are passed to JVM), however every integer is a real number (see [Numerical tower](https://en.wikipedia.org/wiki/Numerical_tower) and, albeit not that useful in practice, [PEP 3141](https://www.python.org/dev/peps/pep-3141/)).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-962420259
**[Test build #144953 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144953/testReport)** for PR 34435 at commit [`65294a4`](https://github.com/apache/spark/commit/65294a4bb2db9a65fa3a0144f583b554a943fdbe).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-961755067
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-960529896
Ping @HyukjinKwon @zero323 Thanks !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34435:
URL: https://github.com/apache/spark/pull/34435#discussion_r743596823
##########
File path: python/pyspark/statcounter.py
##########
@@ -114,24 +120,24 @@ def variance(self):
# Return the sample variance, which corrects for bias in estimating the variance by dividing
# by N-1 instead of N.
#
- def sampleVariance(self):
+ def sampleVariance(self) -> float:
if self.n <= 1:
return float('nan')
else:
return self.m2 / (self.n - 1)
# Return the standard deviation of the values.
- def stdev(self):
+ def stdev(self) -> float:
return sqrt(self.variance())
#
# Return the sample standard deviation of the values, which corrects for bias in estimating the
# variance by dividing by N-1 instead of N.
#
- def sampleStdev(self):
+ def sampleStdev(self) -> float:
return sqrt(self.sampleVariance())
- def asDict(self, sample=False):
+ def asDict(self, sample: bool = False) -> Dict[str, Union[float, int]]:
Review comment:
There are contexts, where distinguishing between these two is important (for us, it is primarily for arguments which are passed to JVM), however every integer is a real number (see [Numerical tower](https://en.wikipedia.org/wiki/Numerical_tower) and, albeit not that useful in practice, [PEP 3141](https://www.python.org/dev/peps/pep-3141/)).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34435:
URL: https://github.com/apache/spark/pull/34435#discussion_r746438809
##########
File path: python/pyspark/statcounter.py
##########
@@ -19,18 +19,19 @@
import copy
import math
+from typing import cast, Dict, Iterable, Optional, Union
try:
from numpy import maximum, minimum, sqrt
except ImportError:
- maximum = max
- minimum = min
- sqrt = math.sqrt
+ maximum = max # type: ignore
+ minimum = min # type: ignore
+ sqrt = math.sqrt # type: ignore
Review comment:
I don't have strong option on choosing specific option, but:
- If we're going to use typed definitions before `try` / `catch` block, we can probably do better than `Any`.
- If we're going to use ignores, these should use specific codes (`[assignment]` if I am not mistaken).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34435:
URL: https://github.com/apache/spark/pull/34435#discussion_r743400255
##########
File path: python/pyspark/statcounter.py
##########
@@ -54,12 +59,13 @@ def merge(self, value):
return self
# Merge another StatCounter into this one, adding up the internal statistics.
- def mergeStats(self, other):
+ def mergeStats(self, other: "StatCounter") -> "StatCounter":
if not isinstance(other, StatCounter):
raise TypeError("Can only merge StatCounter but got %s" % type(other))
if other is self: # reference equality holds
- self.merge(copy.deepcopy(other)) # Avoid overwriting fields in a weird order
+ self.merge(
+ copy.deepcopy(cast(float, other))) # Avoid overwriting fields in a weird order
Review comment:
this looks weird. when other is self, shouldn't `other` be `StatCounter`?
##########
File path: python/pyspark/statcounter.py
##########
@@ -19,6 +19,11 @@
import copy
import math
+from typing import cast, Any, Dict, Iterable, Optional, Union
+
+maximum: Any
+minimum: Any
+sqrt: Any
Review comment:
can we avoid this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-962426321
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144953/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-962424992
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49424/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-971118846
**[Test build #145303 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145303/testReport)** for PR 34435 at commit [`5488504`](https://github.com/apache/spark/commit/54885041a7beceeeb170431482b9fe9dab976bfb).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-971139617
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49773/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-961755067
**[Test build #144929 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144929/testReport)** for PR 34435 at commit [`5fdb758`](https://github.com/apache/spark/commit/5fdb75825f1cbf6635e21455bfd4fcc20a583b3d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-972682263
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49854/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 closed pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
zero323 closed pull request #34435:
URL: https://github.com/apache/spark/pull/34435
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-972609689
**[Test build #145381 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145381/testReport)** for PR 34435 at commit [`44ffdaf`](https://github.com/apache/spark/commit/44ffdaf649d2b958085176f151b91dbcdbad4b3b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34435:
URL: https://github.com/apache/spark/pull/34435#discussion_r743476830
##########
File path: python/pyspark/statcounter.py
##########
@@ -114,24 +120,24 @@ def variance(self):
# Return the sample variance, which corrects for bias in estimating the variance by dividing
# by N-1 instead of N.
#
- def sampleVariance(self):
+ def sampleVariance(self) -> float:
if self.n <= 1:
return float('nan')
else:
return self.m2 / (self.n - 1)
# Return the standard deviation of the values.
- def stdev(self):
+ def stdev(self) -> float:
return sqrt(self.variance())
#
# Return the sample standard deviation of the values, which corrects for bias in estimating the
# variance by dividing by N-1 instead of N.
#
- def sampleStdev(self):
+ def sampleStdev(self) -> float:
return sqrt(self.sampleVariance())
- def asDict(self, sample=False):
+ def asDict(self, sample: bool = False) -> Dict[str, Union[float, int]]:
Review comment:
@zero323 I think we should keep ```Dict[str, Union[float, int]]``` because of ```'count': self.count()``` is ```int``` in the return. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34435:
URL: https://github.com/apache/spark/pull/34435#discussion_r743476830
##########
File path: python/pyspark/statcounter.py
##########
@@ -114,24 +120,24 @@ def variance(self):
# Return the sample variance, which corrects for bias in estimating the variance by dividing
# by N-1 instead of N.
#
- def sampleVariance(self):
+ def sampleVariance(self) -> float:
if self.n <= 1:
return float('nan')
else:
return self.m2 / (self.n - 1)
# Return the standard deviation of the values.
- def stdev(self):
+ def stdev(self) -> float:
return sqrt(self.variance())
#
# Return the sample standard deviation of the values, which corrects for bias in estimating the
# variance by dividing by N-1 instead of N.
#
- def sampleStdev(self):
+ def sampleStdev(self) -> float:
return sqrt(self.sampleVariance())
- def asDict(self, sample=False):
+ def asDict(self, sample: bool = False) -> Dict[str, Union[float, int]]:
Review comment:
@zero323 I think we should keep ```Dict[str, Union[float, int]]``` because of ```'count': self.count()``` is ```int``` in the return. WDYT?
##########
File path: python/pyspark/statcounter.py
##########
@@ -19,18 +19,19 @@
import copy
import math
+from typing import cast, Dict, Iterable, Optional, Union
try:
from numpy import maximum, minimum, sqrt
except ImportError:
- maximum = max
- minimum = min
- sqrt = math.sqrt
+ maximum = max # type: ignore
+ minimum = min # type: ignore
+ sqrt = math.sqrt # type: ignore
Review comment:
@HyukjinKwon, as the issue [#1153](https://github.com/python/mypy/issues/1153) in mypy, I think we should ignore this. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-971085411
**[Test build #145298 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145298/testReport)** for PR 34435 at commit [`8752dff`](https://github.com/apache/spark/commit/8752dff05af04776e3a8f0fb17c9d67fffdd8395).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class SparkConf(object):`
* `class ProbabilisticClassifier(Classifier, _ProbabilisticClassifierParams, metaclass=ABCMeta):`
* `class ProbabilisticClassificationModel(`
* `class _JavaProbabilisticClassifier(ProbabilisticClassifier, _JavaClassifier, metaclass=ABCMeta):`
* `class _JavaProbabilisticClassificationModel(`
* `class _LinearSVCParams(`
* `class LinearSVCModel(`
* `class _LogisticRegressionParams(`
* `class LogisticRegression(`
* `class LogisticRegressionModel(`
* `class BinaryLogisticRegressionSummary(_BinaryClassificationSummary, LogisticRegressionSummary):`
* `class BinaryLogisticRegressionTrainingSummary(`
* `class DecisionTreeClassifier(`
* `class DecisionTreeClassificationModel(`
* `class RandomForestClassifier(`
* `class RandomForestClassificationModel(`
* `class RandomForestClassificationTrainingSummary(`
* `class BinaryRandomForestClassificationTrainingSummary(`
* `class GBTClassifier(`
* `class GBTClassificationModel(`
* `class NaiveBayes(`
* `class NaiveBayesModel(`
* `class _MultilayerPerceptronParams(`
* `class MultilayerPerceptronClassifier(`
* `class MultilayerPerceptronClassificationModel(`
* `class MultilayerPerceptronClassificationTrainingSummary(`
* `class FMClassifier(`
* `class FMClassificationModel(`
* `class _GaussianMixtureParams(`
* `class GaussianMixtureModel(`
* `class _KMeansParams(`
* `class KMeansModel(`
* `class _BisectingKMeansParams(`
* `class BisectingKMeansModel(`
* `class PowerIterationClustering(`
* `class BinaryClassificationEvaluator(`
* `class RegressionEvaluator(`
* `class MulticlassClassificationEvaluator(`
* `class MultilabelClassificationEvaluator(`
* `class ClusteringEvaluator(`
* `class RankingEvaluator(`
* `class Binarizer(`
* `class BucketedRandomProjectionLSH(`
* `class BucketedRandomProjectionLSHModel(`
* `class Bucketizer(`
* `class ElementwiseProduct(`
* `class FeatureHasher(`
* `class HashingTF(`
* `class _OneHotEncoderParams(`
* `class PolynomialExpansion(`
* `class QuantileDiscretizer(`
* `class _StringIndexerParams(`
* `class StopWordsRemover(`
* `class VectorAssembler(`
* `class VectorSizeHint(`
* `class VarianceThresholdSelector(`
* `class VarianceThresholdSelectorModel(`
* `class UnivariateFeatureSelector(`
* `class UnivariateFeatureSelectorModel(`
* `class _LinearRegressionParams(`
* `class LinearRegressionModel(`
* `class IsotonicRegression(`
* `class IsotonicRegressionModel(JavaModel, _IsotonicRegressionParams, JavaMLWritable, JavaMLReadable):`
* `class DecisionTreeRegressor(`
* `class RandomForestRegressor(`
* `class _AFTSurvivalRegressionParams(`
* `class AFTSurvivalRegression(`
* `class AFTSurvivalRegressionModel(`
* `class _GeneralizedLinearRegressionParams(`
* `class GeneralizedLinearRegression(`
* `class GeneralizedLinearRegressionModel(`
* `class _FactorizationMachinesParams(`
* `class FMRegressionModel(`
* `class CrossValidator(`
* `class TrainValidationSplit(`
* ` + \"class name `
* `class MultivariateGaussian(NamedTuple):`
* `class ResultIterable(Iterable[T]):`
* `trait ExpressionBuilder `
* `case class ExpressionStats(expr: Expression)(var useCount: Int)`
* `trait PadExpressionBuilderBase extends ExpressionBuilder `
* `case class StringLPad(str: Expression, len: Expression, pad: Expression)`
* `case class BinaryLPad(str: Expression, len: Expression, pad: Expression, child: Expression)`
* `case class BinaryRPad(str: Expression, len: Expression, pad: Expression, child: Expression)`
* `case class DropIndex(`
* `public class ColumnIOUtil `
* `case class ParquetColumn(`
* `case class DropIndexExec(`
* `case class PushedDownOperators(`
* `case class TableSampleInfo(`
* `class RatePerMicroBatchProvider extends SimpleTableProvider with DataSourceRegister `
* `class RatePerMicroBatchTable(`
* `class RatePerMicroBatchStream(`
* `case class RatePerMicroBatchStreamOffset(offset: Long, timestamp: Long) extends Offset `
* `case class RatePerMicroBatchStreamInputPartition(`
* `class RatePerMicroBatchStreamPartitionReader(`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-971118846
**[Test build #145303 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145303/testReport)** for PR 34435 at commit [`5488504`](https://github.com/apache/spark/commit/54885041a7beceeeb170431482b9fe9dab976bfb).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-971132249
**[Test build #145303 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145303/testReport)** for PR 34435 at commit [`5488504`](https://github.com/apache/spark/commit/54885041a7beceeeb170431482b9fe9dab976bfb).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-971172254
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49773/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-966045528
**[Test build #145093 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145093/testReport)** for PR 34435 at commit [`8ba331b`](https://github.com/apache/spark/commit/8ba331b1c25e1c87e65333696fc001cc7312b6e8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-966172141
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49562/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-961830523
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49400/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-961791371
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34435:
URL: https://github.com/apache/spark/pull/34435#discussion_r743476830
##########
File path: python/pyspark/statcounter.py
##########
@@ -114,24 +120,24 @@ def variance(self):
# Return the sample variance, which corrects for bias in estimating the variance by dividing
# by N-1 instead of N.
#
- def sampleVariance(self):
+ def sampleVariance(self) -> float:
if self.n <= 1:
return float('nan')
else:
return self.m2 / (self.n - 1)
# Return the standard deviation of the values.
- def stdev(self):
+ def stdev(self) -> float:
return sqrt(self.variance())
#
# Return the sample standard deviation of the values, which corrects for bias in estimating the
# variance by dividing by N-1 instead of N.
#
- def sampleStdev(self):
+ def sampleStdev(self) -> float:
return sqrt(self.sampleVariance())
- def asDict(self, sample=False):
+ def asDict(self, sample: bool = False) -> Dict[str, Union[float, int]]:
Review comment:
@zero323 I think we should keep ```Dict[str, Union[float, int]]``` because of ```'count': self.count()``` is ```int``` in the return. WDYT?
##########
File path: python/pyspark/statcounter.py
##########
@@ -19,18 +19,19 @@
import copy
import math
+from typing import cast, Dict, Iterable, Optional, Union
try:
from numpy import maximum, minimum, sqrt
except ImportError:
- maximum = max
- minimum = min
- sqrt = math.sqrt
+ maximum = max # type: ignore
+ minimum = min # type: ignore
+ sqrt = math.sqrt # type: ignore
Review comment:
@HyukjinKwon, as the issue [#1153](https://github.com/python/mypy/issues/1153) in mypy, I think we should ignore this. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-962420259
**[Test build #144953 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144953/testReport)** for PR 34435 at commit [`65294a4`](https://github.com/apache/spark/commit/65294a4bb2db9a65fa3a0144f583b554a943fdbe).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34435:
URL: https://github.com/apache/spark/pull/34435#discussion_r747950440
##########
File path: python/pyspark/statcounter.py
##########
@@ -19,18 +19,19 @@
import copy
import math
+from typing import cast, Dict, Iterable, Optional, Union
try:
from numpy import maximum, minimum, sqrt
except ImportError:
- maximum = max
- minimum = min
- sqrt = math.sqrt
+ maximum = max # type: ignore
+ minimum = min # type: ignore
+ sqrt = math.sqrt # type: ignore
Review comment:
@zero323 , can you review it again? Many thanks 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-966120320
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49562/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-969498400
im gonna leave it to @zero323 since he has been reviewing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-972630796
**[Test build #145381 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145381/testReport)** for PR 34435 at commit [`44ffdaf`](https://github.com/apache/spark/commit/44ffdaf649d2b958085176f151b91dbcdbad4b3b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-972730227
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49854/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34435:
URL: https://github.com/apache/spark/pull/34435#discussion_r751357113
##########
File path: python/pyspark/statcounter.py
##########
@@ -53,12 +54,12 @@ def merge(self, value):
return self
# Merge another StatCounter into this one, adding up the internal statistics.
- def mergeStats(self, other):
+ def mergeStats(self, other: "StatCounter") -> "StatCounter":
if not isinstance(other, StatCounter):
raise TypeError("Can only merge StatCounter but got %s" % type(other))
if other is self: # reference equality holds
- self.merge(copy.deepcopy(other)) # Avoid overwriting fields in a weird order
+ self.mergeStats(copy.deepcopy(other)) # Avoid overwriting fields in a weird order
Review comment:
Let's `type: ignore` handle this as a separate ticket. This has been here forever and was never reported, so there is good indication it is a piece of dead code, and we should cover this scenario with unit tests anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-954543196
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144742/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-961793909
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49400/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-961776815
**[Test build #144929 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144929/testReport)** for PR 34435 at commit [`5fdb758`](https://github.com/apache/spark/commit/5fdb75825f1cbf6635e21455bfd4fcc20a583b3d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-961755067
**[Test build #144929 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144929/testReport)** for PR 34435 at commit [`5fdb758`](https://github.com/apache/spark/commit/5fdb75825f1cbf6635e21455bfd4fcc20a583b3d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-961791371
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-962425146
This PR is ready to review again! Thanks, @zero323 @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-954543196
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144742/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-954563245
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49211/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-966168921
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49562/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-966045528
**[Test build #145093 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145093/testReport)** for PR 34435 at commit [`8ba331b`](https://github.com/apache/spark/commit/8ba331b1c25e1c87e65333696fc001cc7312b6e8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-972740493
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49854/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-972778919
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-971117271
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145298/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-971041767
**[Test build #145298 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145298/testReport)** for PR 34435 at commit [`8752dff`](https://github.com/apache/spark/commit/8752dff05af04776e3a8f0fb17c9d67fffdd8395).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-972782779
Thanks all !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-971142237
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
ueshin commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-973310737
I submitted a PR to fix the issue https://github.com/apache/spark/pull/34435#discussion_r750668911 at #34653
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-972609689
**[Test build #145381 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145381/testReport)** for PR 34435 at commit [`44ffdaf`](https://github.com/apache/spark/commit/44ffdaf649d2b958085176f151b91dbcdbad4b3b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-968737803
cc @zero323 , @HyukjinKwon , @ueshin !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-971172254
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49773/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-966172141
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49562/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-971117271
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145298/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-971160905
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49773/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-954613242
CC @HyukjinKwon @zero323 @ueshin FYI. Many thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-972645595
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145381/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34435:
URL: https://github.com/apache/spark/pull/34435#discussion_r743501203
##########
File path: python/pyspark/statcounter.py
##########
@@ -19,18 +19,19 @@
import copy
import math
+from typing import cast, Dict, Iterable, Optional, Union
try:
from numpy import maximum, minimum, sqrt
except ImportError:
- maximum = max
- minimum = min
- sqrt = math.sqrt
+ maximum = max # type: ignore
+ minimum = min # type: ignore
+ sqrt = math.sqrt # type: ignore
Review comment:
@HyukjinKwon, as the issue [#1153](https://github.com/python/mypy/issues/1153) in mypy, I think we should ignore this. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34435:
URL: https://github.com/apache/spark/pull/34435#discussion_r743453323
##########
File path: python/pyspark/statcounter.py
##########
@@ -114,24 +120,24 @@ def variance(self):
# Return the sample variance, which corrects for bias in estimating the variance by dividing
# by N-1 instead of N.
#
- def sampleVariance(self):
+ def sampleVariance(self) -> float:
if self.n <= 1:
return float('nan')
else:
return self.m2 / (self.n - 1)
# Return the standard deviation of the values.
- def stdev(self):
+ def stdev(self) -> float:
return sqrt(self.variance())
#
# Return the sample standard deviation of the values, which corrects for bias in estimating the
# variance by dividing by N-1 instead of N.
#
- def sampleStdev(self):
+ def sampleStdev(self) -> float:
return sqrt(self.sampleVariance())
- def asDict(self, sample=False):
+ def asDict(self, sample: bool = False) -> Dict[str, Union[float, int]]:
Review comment:
Nit. Might be cleaner to have `Dict[str, float]`.
##########
File path: python/pyspark/statcounter.py
##########
@@ -114,24 +120,24 @@ def variance(self):
# Return the sample variance, which corrects for bias in estimating the variance by dividing
# by N-1 instead of N.
#
- def sampleVariance(self):
+ def sampleVariance(self) -> float:
if self.n <= 1:
return float('nan')
else:
return self.m2 / (self.n - 1)
# Return the standard deviation of the values.
- def stdev(self):
+ def stdev(self) -> float:
return sqrt(self.variance())
#
# Return the sample standard deviation of the values, which corrects for bias in estimating the
# variance by dividing by N-1 instead of N.
#
- def sampleStdev(self):
+ def sampleStdev(self) -> float:
return sqrt(self.sampleVariance())
- def asDict(self, sample=False):
+ def asDict(self, sample: bool = False) -> Dict[str, Union[float, int]]:
Review comment:
There are contexts, where distinguishing between these two is important (for us, it is primarily for arguments which are passed to JVM), however every integer is a real number (see [Numerical tower](https://en.wikipedia.org/wiki/Numerical_tower) and, albeit not that useful in practice, [PEP 3141](https://www.python.org/dev/peps/pep-3141/)).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34435:
URL: https://github.com/apache/spark/pull/34435#discussion_r743453323
##########
File path: python/pyspark/statcounter.py
##########
@@ -114,24 +120,24 @@ def variance(self):
# Return the sample variance, which corrects for bias in estimating the variance by dividing
# by N-1 instead of N.
#
- def sampleVariance(self):
+ def sampleVariance(self) -> float:
if self.n <= 1:
return float('nan')
else:
return self.m2 / (self.n - 1)
# Return the standard deviation of the values.
- def stdev(self):
+ def stdev(self) -> float:
return sqrt(self.variance())
#
# Return the sample standard deviation of the values, which corrects for bias in estimating the
# variance by dividing by N-1 instead of N.
#
- def sampleStdev(self):
+ def sampleStdev(self) -> float:
return sqrt(self.sampleVariance())
- def asDict(self, sample=False):
+ def asDict(self, sample: bool = False) -> Dict[str, Union[float, int]]:
Review comment:
Nit. Might be cleaner to have `Dict[str, float]`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-961698191
In general, this should be rewritten with `Protocols` or equivalent at some point, but it is not priority and it is not in the scope of this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34435:
URL: https://github.com/apache/spark/pull/34435#discussion_r743400255
##########
File path: python/pyspark/statcounter.py
##########
@@ -54,12 +59,13 @@ def merge(self, value):
return self
# Merge another StatCounter into this one, adding up the internal statistics.
- def mergeStats(self, other):
+ def mergeStats(self, other: "StatCounter") -> "StatCounter":
if not isinstance(other, StatCounter):
raise TypeError("Can only merge StatCounter but got %s" % type(other))
if other is self: # reference equality holds
- self.merge(copy.deepcopy(other)) # Avoid overwriting fields in a weird order
+ self.merge(
+ copy.deepcopy(cast(float, other))) # Avoid overwriting fields in a weird order
Review comment:
this looks weird. when other is self, shouldn't `other` be `StatCounter`?
##########
File path: python/pyspark/statcounter.py
##########
@@ -19,6 +19,11 @@
import copy
import math
+from typing import cast, Any, Dict, Iterable, Optional, Union
+
+maximum: Any
+minimum: Any
+sqrt: Any
Review comment:
can we avoid this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-961791371
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144929/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #34435:
URL: https://github.com/apache/spark/pull/34435#issuecomment-961698191
In general, this should be rewritten with `Protocols` or equivalent at some point, but it is not priority and it is not in the scope of this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34435: [SPARK-37155][PYTHON] Inline type hints for python/pyspark/statcounter.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34435:
URL: https://github.com/apache/spark/pull/34435#discussion_r745226496
##########
File path: python/pyspark/statcounter.py
##########
@@ -19,6 +19,11 @@
import copy
import math
+from typing import cast, Any, Dict, Iterable, Optional, Union
+
+maximum: Any
+minimum: Any
+sqrt: Any
Review comment:
updated, https://github.com/apache/spark/pull/34435#discussion_r743501203. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org