You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/11/08 06:48:38 UTC
[GitHub] [spark] dchvn opened a new pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
dchvn opened a new pull request #34513:
URL: https://github.com/apache/spark/pull/34513
### What changes were proposed in this pull request?
Inline type hints for python/pyspark/mllib/stat/_statistics.py
### Why are the changes needed?
We can take advantage of static type checking within the functions by inlining the type hints.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Existing tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-975272886
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49976/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-977454201
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50026/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-962916414
**[Test build #144992 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144992/testReport)** for PR 34513 at commit [`986f406`](https://github.com/apache/spark/commit/986f4064d82de0b975ccd8170ad05a1aa2cacc61).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-965550421
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49536/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-965994673
**[Test build #145081 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145081/testReport)** for PR 34513 at commit [`e146abe`](https://github.com/apache/spark/commit/e146abec8bf19d28b25b037c6ce84b128e250281).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34513:
URL: https://github.com/apache/spark/pull/34513#discussion_r753784172
##########
File path: python/pyspark/mllib/stat/_statistics.py
##########
@@ -170,10 +190,29 @@ def corr(x, y=None, method=None):
if not y:
return callMLlibFunc("corr", x.map(_convert_to_vector), method).toArray()
else:
- return callMLlibFunc("corr", x.map(float), y.map(float), method)
+ return callMLlibFunc(
+ "corr", x.map(float), y.map(float), method # type: ignore[arg-type]
Review comment:
`float` is `Callable`. What is the exact error you get here if it is not ignored.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34513:
URL: https://github.com/apache/spark/pull/34513#discussion_r753783981
##########
File path: python/pyspark/mllib/linalg/__init__.pyi
##########
@@ -68,6 +68,7 @@ class Vector:
__UDT__: VectorUDT
def toArray(self) -> ndarray: ...
def asML(self) -> newlinalg.Vector: ...
+ def __len__(self) -> int: ...
Review comment:
This doesn't look right. ATM `Vector` has not `__len__` method (only subclassess do), so shouldn't have annotation for it.
I believe we should `type: ignore` this for now, and decide if and how to adjust parent class definitions separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-978815088
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50076/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34513:
URL: https://github.com/apache/spark/pull/34513#discussion_r756781924
##########
File path: python/pyspark/mllib/stat/_statistics.py
##########
@@ -168,12 +190,34 @@ def corr(x, y=None, method=None):
raise TypeError("Use 'method=' to specify method name.")
if not y:
- return callMLlibFunc("corr", x.map(_convert_to_vector), method).toArray()
+ return cast(
+ JavaObject, callMLlibFunc("corr", x.map(_convert_to_vector), method)
+ ).toArray()
else:
- return callMLlibFunc("corr", x.map(float), y.map(float), method)
+ return cast(
+ Union[float, Matrix],
Review comment:
That's not `Union`. If you are in this specific branch, you have two `RDD[float]` and result should be float.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-979653432
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145640/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-975316063
**[Test build #145506 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145506/testReport)** for PR 34513 at commit [`784a148`](https://github.com/apache/spark/commit/784a1484e861dcd2670cabd02692dac36f00d113).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34513:
URL: https://github.com/apache/spark/pull/34513#discussion_r753786802
##########
File path: python/pyspark/mllib/stat/_statistics.py
##########
@@ -16,51 +16,57 @@
#
import sys
+from typing import overload, List, Optional, Union
+from typing_extensions import Literal
Review comment:
We should never have `typing_extensions` extensions imported directly in `py` module. Let's move this and `CorrelationMethod` to `_typing.pyi`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-975215676
**[Test build #145504 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145504/testReport)** for PR 34513 at commit [`836903a`](https://github.com/apache/spark/commit/836903af42aae90e333ee04af7ae1170bcfbce34).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-962861356
**[Test build #144986 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144986/testReport)** for PR 34513 at commit [`986f406`](https://github.com/apache/spark/commit/986f4064d82de0b975ccd8170ad05a1aa2cacc61).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-966015946
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145081/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-977393918
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50026/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-979602119
**[Test build #145640 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145640/testReport)** for PR 34513 at commit [`0df579e`](https://github.com/apache/spark/commit/0df579e7df596f8b960b5716a2959f8b481763b7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-975215676
**[Test build #145504 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145504/testReport)** for PR 34513 at commit [`836903a`](https://github.com/apache/spark/commit/836903af42aae90e333ee04af7ae1170bcfbce34).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34513:
URL: https://github.com/apache/spark/pull/34513#discussion_r755596954
##########
File path: python/pyspark/mllib/stat/_statistics.py
##########
@@ -170,10 +190,29 @@ def corr(x, y=None, method=None):
if not y:
return callMLlibFunc("corr", x.map(_convert_to_vector), method).toArray()
else:
- return callMLlibFunc("corr", x.map(float), y.map(float), method)
+ return callMLlibFunc(
+ "corr", x.map(float), y.map(float), method # type: ignore[arg-type]
Review comment:
In such case it would be better to `cast` `x`, as we expect it is going to be `RDD[float]`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-962901986
**[Test build #144992 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144992/testReport)** for PR 34513 at commit [`986f406`](https://github.com/apache/spark/commit/986f4064d82de0b975ccd8170ad05a1aa2cacc61).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-962936575
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-977393605
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145554/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-979662149
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50111/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-979653432
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145640/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-975232708
**[Test build #145504 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145504/testReport)** for PR 34513 at commit [`836903a`](https://github.com/apache/spark/commit/836903af42aae90e333ee04af7ae1170bcfbce34).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-975318460
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49976/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-965354492
**[Test build #145067 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145067/testReport)** for PR 34513 at commit [`157c380`](https://github.com/apache/spark/commit/157c380a4c4e783e6ba25f6223ee33e78ec92451).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-962925106
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49458/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-966012139
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49550/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-975290073
**[Test build #145506 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145506/testReport)** for PR 34513 at commit [`784a148`](https://github.com/apache/spark/commit/784a1484e861dcd2670cabd02692dac36f00d113).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34513:
URL: https://github.com/apache/spark/pull/34513#discussion_r753995410
##########
File path: python/pyspark/mllib/stat/_statistics.py
##########
@@ -170,10 +190,29 @@ def corr(x, y=None, method=None):
if not y:
return callMLlibFunc("corr", x.map(_convert_to_vector), method).toArray()
else:
- return callMLlibFunc("corr", x.map(float), y.map(float), method)
+ return callMLlibFunc(
+ "corr", x.map(float), y.map(float), method # type: ignore[arg-type]
Review comment:
```python
Argument 1 to "map" of "RDD" has incompatible type "Type[float]"; expected "Callable[[Vector], Any]" [arg-type]
```
I got that error if it is not ignored, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34513:
URL: https://github.com/apache/spark/pull/34513#discussion_r753808683
##########
File path: python/pyspark/mllib/linalg/__init__.pyi
##########
@@ -68,6 +68,7 @@ class Vector:
__UDT__: VectorUDT
def toArray(self) -> ndarray: ...
def asML(self) -> newlinalg.Vector: ...
+ def __len__(self) -> int: ...
Review comment:
For the record SPARK-37431
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-975342596
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34513:
URL: https://github.com/apache/spark/pull/34513#discussion_r756794845
##########
File path: python/pyspark/mllib/stat/_statistics.py
##########
@@ -168,12 +190,34 @@ def corr(x, y=None, method=None):
raise TypeError("Use 'method=' to specify method name.")
if not y:
- return callMLlibFunc("corr", x.map(_convert_to_vector), method).toArray()
+ return cast(
+ JavaObject, callMLlibFunc("corr", x.map(_convert_to_vector), method)
+ ).toArray()
else:
- return callMLlibFunc("corr", x.map(float), y.map(float), method)
+ return cast(
+ Union[float, Matrix],
Review comment:
@nchammas Just looking at this, we might have to make adjust signatures for `callMLlibFunc` and `callMLlibFunc` which match `_java2py` but don't take into account types that are automatically converted to Python types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-978783988
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50076/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-962869742
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144986/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-962901986
**[Test build #144992 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144992/testReport)** for PR 34513 at commit [`986f406`](https://github.com/apache/spark/commit/986f4064d82de0b975ccd8170ad05a1aa2cacc61).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-966004632
**[Test build #145081 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145081/testReport)** for PR 34513 at commit [`e146abe`](https://github.com/apache/spark/commit/e146abec8bf19d28b25b037c6ce84b128e250281).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-966015946
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145081/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34513:
URL: https://github.com/apache/spark/pull/34513#discussion_r747203964
##########
File path: python/pyspark/mllib/stat/_statistics.py
##########
@@ -170,10 +190,29 @@ def corr(x, y=None, method=None):
if not y:
return callMLlibFunc("corr", x.map(_convert_to_vector), method).toArray()
else:
- return callMLlibFunc("corr", x.map(float), y.map(float), method)
+ return callMLlibFunc(
+ "corr", x.map(float), y.map(float), method # type: ignore[arg-type]
Review comment:
```RDD.map``` expect a ```Callable``` argument
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-962995711
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49465/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-977454201
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50026/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34513:
URL: https://github.com/apache/spark/pull/34513#discussion_r756528246
##########
File path: python/pyspark/mllib/stat/_statistics.py
##########
@@ -33,34 +39,34 @@ class MultivariateStatisticalSummary(JavaModelWrapper):
Trait for multivariate statistical summary of a data matrix.
"""
- def mean(self):
- return self.call("mean").toArray()
+ def mean(self) -> ndarray:
+ return cast(JavaObject, self.call("mean")).toArray()
Review comment:
a lot of `cast(JavaObject, self.call("..."))` added because `bytearray` and `bytes` does not have `toArray`
##########
File path: python/pyspark/mllib/stat/_statistics.py
##########
@@ -168,12 +190,34 @@ def corr(x, y=None, method=None):
raise TypeError("Use 'method=' to specify method name.")
if not y:
- return callMLlibFunc("corr", x.map(_convert_to_vector), method).toArray()
+ return cast(
+ JavaObject, callMLlibFunc("corr", x.map(_convert_to_vector), method)
+ ).toArray()
else:
- return callMLlibFunc("corr", x.map(float), y.map(float), method)
+ return cast(
Review comment:
`callMLlibFunc` return `Union[JavaObject, bytearray, bytes]` but we expect `Union[float, Matrix]`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-977393605
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145554/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-979681961
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50111/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-977357074
**[Test build #145554 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145554/testReport)** for PR 34513 at commit [`991fd8c`](https://github.com/apache/spark/commit/991fd8cb8bbb7b33120b754672889ab90b77091a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-978807682
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145610/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-1059976967
This should be resynced with upstream to resolve conflicts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-962995656
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49465/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-963006835
**[Test build #144997 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144997/testReport)** for PR 34513 at commit [`b95c8fa`](https://github.com/apache/spark/commit/b95c8fa0917c41f5cf86312d295e2a46dfa648e3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-963078037
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49468/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-966043899
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49550/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-966039510
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49550/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-965550421
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49536/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-966043899
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49550/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-975404480
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49978/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-975404480
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49978/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34513:
URL: https://github.com/apache/spark/pull/34513#discussion_r753783981
##########
File path: python/pyspark/mllib/linalg/__init__.pyi
##########
@@ -68,6 +68,7 @@ class Vector:
__UDT__: VectorUDT
def toArray(self) -> ndarray: ...
def asML(self) -> newlinalg.Vector: ...
+ def __len__(self) -> int: ...
Review comment:
This doesn't look right. ATM `Vector` has not `__len__` method (only subclassess do), so shouldn't have annotation for it.
I believe we should `type: ignore` this for now, and decide if and how to adjust parent class definitions separately.
##########
File path: python/pyspark/mllib/stat/_statistics.py
##########
@@ -170,10 +190,29 @@ def corr(x, y=None, method=None):
if not y:
return callMLlibFunc("corr", x.map(_convert_to_vector), method).toArray()
else:
- return callMLlibFunc("corr", x.map(float), y.map(float), method)
+ return callMLlibFunc(
+ "corr", x.map(float), y.map(float), method # type: ignore[arg-type]
Review comment:
`float` is `Callable`. What is the exact error you get here if it is not ignored.
##########
File path: python/pyspark/mllib/stat/_statistics.py
##########
@@ -170,10 +190,29 @@ def corr(x, y=None, method=None):
if not y:
return callMLlibFunc("corr", x.map(_convert_to_vector), method).toArray()
else:
- return callMLlibFunc("corr", x.map(float), y.map(float), method)
+ return callMLlibFunc(
+ "corr", x.map(float), y.map(float), method # type: ignore[arg-type]
Review comment:
`float` is `Callable`. What is the exact error you get here if it is not ignored?
##########
File path: python/pyspark/mllib/stat/_statistics.py
##########
@@ -16,51 +16,57 @@
#
import sys
+from typing import overload, List, Optional, Union
+from typing_extensions import Literal
+
+from numpy import ndarray
from pyspark.rdd import RDD
from pyspark.mllib.common import callMLlibFunc, JavaModelWrapper
-from pyspark.mllib.linalg import Matrix, _convert_to_vector
+from pyspark.mllib.linalg import Matrix, Vector, _convert_to_vector # type: ignore[attr-defined]
Review comment:
This seems to indicate that we should annotate `__init__.py` first, but I guess that's too late.
##########
File path: python/pyspark/mllib/stat/_statistics.py
##########
@@ -16,51 +16,57 @@
#
import sys
+from typing import overload, List, Optional, Union
+from typing_extensions import Literal
Review comment:
We should never have `typing_extensions` extensions imported directly in `py` module. Let's move this and `CorrelationMethod` to `_typing.pyi`.
##########
File path: python/pyspark/mllib/linalg/__init__.pyi
##########
@@ -68,6 +68,7 @@ class Vector:
__UDT__: VectorUDT
def toArray(self) -> ndarray: ...
def asML(self) -> newlinalg.Vector: ...
+ def __len__(self) -> int: ...
Review comment:
For the record SPARK-37431
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34513:
URL: https://github.com/apache/spark/pull/34513#discussion_r753786672
##########
File path: python/pyspark/mllib/stat/_statistics.py
##########
@@ -16,51 +16,57 @@
#
import sys
+from typing import overload, List, Optional, Union
+from typing_extensions import Literal
+
+from numpy import ndarray
from pyspark.rdd import RDD
from pyspark.mllib.common import callMLlibFunc, JavaModelWrapper
-from pyspark.mllib.linalg import Matrix, _convert_to_vector
+from pyspark.mllib.linalg import Matrix, Vector, _convert_to_vector # type: ignore[attr-defined]
Review comment:
This seems to indicate that we should annotate `__init__.py` first, but I guess that's too late.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-977445837
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50026/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-979648473
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50111/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-975290073
**[Test build #145506 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145506/testReport)** for PR 34513 at commit [`784a148`](https://github.com/apache/spark/commit/784a1484e861dcd2670cabd02692dac36f00d113).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-975352297
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49978/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-975342596
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-963803717
CC @HyukjinKwon @zero323 FYI. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34513:
URL: https://github.com/apache/spark/pull/34513#discussion_r744445424
##########
File path: python/pyspark/mllib/stat/_statistics.py
##########
@@ -171,10 +194,31 @@ def corr(x, y=None, method=None):
if not y:
return callMLlibFunc("corr", x.map(_convert_to_vector), method).toArray()
else:
- return callMLlibFunc("corr", x.map(float), y.map(float), method)
+ return callMLlibFunc(
+ "corr", x.map(float), y.map(float), method) # type: ignore[arg-type]
Review comment:
```RDD.map``` expect a ```Callable``` argument
##########
File path: python/pyspark/mllib/linalg/__init__.pyi
##########
@@ -68,6 +68,7 @@ class Vector:
__UDT__: VectorUDT
def toArray(self) -> ndarray: ...
def asML(self) -> newlinalg.Vector: ...
+ def __len__(self) -> int: ...
Review comment:
```__len__``` is added to match with logic in
https://github.com/apache/spark/blob/e29c4e12fbaba542e8150fb71e1c0c1fc15813bd/python/pyspark/mllib/stat/_statistics.py#L268
##########
File path: python/pyspark/mllib/stat/_statistics.py
##########
@@ -335,12 +381,12 @@ def kolmogorovSmirnovTest(data, distName="norm", *params):
if not isinstance(distName, str):
raise TypeError("distName should be a string, got %s." % type(distName))
- params = [float(param) for param in params]
+ param_list = [float(param) for param in params]
Review comment:
```param_list``` is intermediate variable, because ```params``` is ```Tuple```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-962900967
Jenkins, retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-962986518
**[Test build #144997 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144997/testReport)** for PR 34513 at commit [`b95c8fa`](https://github.com/apache/spark/commit/b95c8fa0917c41f5cf86312d295e2a46dfa648e3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-963025291
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49468/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-963032633
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144997/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-978807682
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145610/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-978815088
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50076/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-978777736
**[Test build #145610 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145610/testReport)** for PR 34513 at commit [`c193c06`](https://github.com/apache/spark/commit/c193c069480fd4aa30c26bfa00c1af28d84b8de4).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-978786837
**[Test build #145610 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145610/testReport)** for PR 34513 at commit [`c193c06`](https://github.com/apache/spark/commit/c193c069480fd4aa30c26bfa00c1af28d84b8de4).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-965403846
**[Test build #145067 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145067/testReport)** for PR 34513 at commit [`157c380`](https://github.com/apache/spark/commit/157c380a4c4e783e6ba25f6223ee33e78ec92451).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-979637964
**[Test build #145640 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145640/testReport)** for PR 34513 at commit [`0df579e`](https://github.com/apache/spark/commit/0df579e7df596f8b960b5716a2959f8b481763b7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-962893869
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49458/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-963078037
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49468/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-962947863
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49465/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-962986518
**[Test build #144997 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144997/testReport)** for PR 34513 at commit [`b95c8fa`](https://github.com/apache/spark/commit/b95c8fa0917c41f5cf86312d295e2a46dfa648e3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-963058332
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49468/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-965480365
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145067/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-965515887
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49536/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-965354492
**[Test build #145067 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145067/testReport)** for PR 34513 at commit [`157c380`](https://github.com/apache/spark/commit/157c380a4c4e783e6ba25f6223ee33e78ec92451).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-975287938
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145504/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34513:
URL: https://github.com/apache/spark/pull/34513#discussion_r754001496
##########
File path: python/pyspark/mllib/stat/_statistics.py
##########
@@ -16,51 +16,57 @@
#
import sys
+from typing import overload, List, Optional, Union
+from typing_extensions import Literal
Review comment:
thanks! updated. I also move
```python
DistName = Literal["norm"]
```
to ```_typing.pyi```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34513:
URL: https://github.com/apache/spark/pull/34513#discussion_r755622882
##########
File path: python/pyspark/mllib/stat/_statistics.py
##########
@@ -170,10 +190,29 @@ def corr(x, y=None, method=None):
if not y:
return callMLlibFunc("corr", x.map(_convert_to_vector), method).toArray()
else:
- return callMLlibFunc("corr", x.map(float), y.map(float), method)
+ return callMLlibFunc(
+ "corr", x.map(float), y.map(float), method # type: ignore[arg-type]
Review comment:
thank you! updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-977381923
**[Test build #145554 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145554/testReport)** for PR 34513 at commit [`991fd8c`](https://github.com/apache/spark/commit/991fd8cb8bbb7b33120b754672889ab90b77091a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-977357074
**[Test build #145554 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145554/testReport)** for PR 34513 at commit [`991fd8c`](https://github.com/apache/spark/commit/991fd8cb8bbb7b33120b754672889ab90b77091a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-978777736
**[Test build #145610 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145610/testReport)** for PR 34513 at commit [`c193c06`](https://github.com/apache/spark/commit/c193c069480fd4aa30c26bfa00c1af28d84b8de4).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-979681961
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50111/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34513:
URL: https://github.com/apache/spark/pull/34513#discussion_r820245641
##########
File path: python/pyspark/mllib/_typing.pyi
##########
@@ -17,10 +17,16 @@
# under the License.
from typing import List, Tuple, TypeVar, Union
+
+from typing_extensions import Literal
+
from pyspark.mllib.linalg import Vector
from numpy import ndarray # noqa: F401
from py4j.java_gateway import JavaObject
VectorLike = Union[ndarray, Vector, List[float], Tuple[float, ...]]
C = TypeVar("C", bound=type)
JavaObjectOrPickleDump = Union[JavaObject, bytearray, bytes]
+
+CorrelationMethod = Union[Literal["spearman"], Literal["pearson"]]
+DistName = Literal["norm"]
Review comment:
Let's use the same convention as for `pyspark.ml` constants.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-965444832
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49536/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-965480365
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145067/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-965994673
**[Test build #145081 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145081/testReport)** for PR 34513 at commit [`e146abe`](https://github.com/apache/spark/commit/e146abec8bf19d28b25b037c6ce84b128e250281).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-965917691
Ping @HyukjinKwon @zero323 Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-978815070
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50076/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zero323 commented on a change in pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #34513:
URL: https://github.com/apache/spark/pull/34513#discussion_r753784312
##########
File path: python/pyspark/mllib/stat/_statistics.py
##########
@@ -170,10 +190,29 @@ def corr(x, y=None, method=None):
if not y:
return callMLlibFunc("corr", x.map(_convert_to_vector), method).toArray()
else:
- return callMLlibFunc("corr", x.map(float), y.map(float), method)
+ return callMLlibFunc(
+ "corr", x.map(float), y.map(float), method # type: ignore[arg-type]
Review comment:
`float` is `Callable`. What is the exact error you get here if it is not ignored?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-975287938
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145504/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-975404412
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49978/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34513:
URL: https://github.com/apache/spark/pull/34513#discussion_r753999742
##########
File path: python/pyspark/mllib/linalg/__init__.pyi
##########
@@ -68,6 +68,7 @@ class Vector:
__UDT__: VectorUDT
def toArray(self) -> ndarray: ...
def asML(self) -> newlinalg.Vector: ...
+ def __len__(self) -> int: ...
Review comment:
thanks! updated with ```ignore```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-979602119
**[Test build #145640 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145640/testReport)** for PR 34513 at commit [`0df579e`](https://github.com/apache/spark/commit/0df579e7df596f8b960b5716a2959f8b481763b7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-962861356
**[Test build #144986 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144986/testReport)** for PR 34513 at commit [`986f406`](https://github.com/apache/spark/commit/986f4064d82de0b975ccd8170ad05a1aa2cacc61).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-962869445
**[Test build #144986 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144986/testReport)** for PR 34513 at commit [`986f406`](https://github.com/apache/spark/commit/986f4064d82de0b975ccd8170ad05a1aa2cacc61).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-962869742
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34513: [SPARK-37234][PYTHON] Inline type hints for python/pyspark/mllib/stat/_statistics.py
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34513:
URL: https://github.com/apache/spark/pull/34513#issuecomment-962995711
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org